Pathogenicity Prediction

Main content

Dominik G. Grimm, Chloe-Agathe Azencott, Fabian Aicheler, Udo Gieraths, et al., and Karsten M. Borgwardt

The evaluation of tools used to predict the impact of missense variants is hindered by two types of circularity


Prioritizing missense variants for further experimental investigation is a key challenge in current sequencing studies for exploring complex diseases. A large number of in silico tools have been employed for this task of pathogenicity prediction, including PolyPhen-2, SIFT, FatHMM, MutationTaster-2, MutationAssessor, CADD, LRT, phyloP and GERP++. Due to the wealth of these methods, an important practical question to answer is which of these tools generalize best, that is, correctly predict the pathogenic character of new variants.

We here demonstrate in a study of 10 tools on five datasets that such a comparative evaluation of pathogenicity prediction tools is hindered by two types of circularity: they arise due to (1) the same variants or (2) different variants from the same protein occurring both in the datasets used for training and for evaluation of these tools, which may lead to overly optimistic results. We show that comparative evaluations of predictors that do not address these types of circularity may erroneously conclude that circularity-confounded tools are most accurate among all tools, and may even outperform optimized combinations of tools, such as Condel and Logit.

Data and Code

Benchmark datasets in Excel format. Contains five tabs, each tab contains the data and the predictied labels and scores from each tool: Benchmark Dataset (XLSX, 23.4 MB)

Supporting Data S1, contains the data in CSV format and all Python scripts to reproduce the Figures, Tables and Supporting Figures from this study: Supporting Data S1 and Python Scripts (ZIP, 6.7 MB)

Data is also available at the VariBench website: VariBench Database for Variations

The code is also available in our GitHub repository here.


Keyboard navigation between tabs via Alt arrow keys as well as Home and End.

Dominik Grimm, Chloé-Agathe Azencott, Fabian Aicheler, Udo Gieraths, Daniel G. MacArthur, Kaitlin E. Samocha, David N. Cooper, Peter D. Stenson, Mark J. Daly, Jordan W. Smoller, Laramie E. Duncan and Karsten Borgwardt
The evaluation of tools used to predict the impact of missense variants is hindered by two types of circularity,
Human Mutation 2015. 36(5):513-523. (Online)

@article {Grimm_HUMU22768,
author = {Grimm, Dominik G. and Azencott, Chlo\'{e}-Agathe and Aicheler, Fabian and Gieraths, Udo and MacArthur, Daniel G. and Samocha, Kaitlin E. and Cooper, David N. and Stenson, Peter D. and Daly, Mark J. and Smoller, Jordan W. and Duncan, Laramie E. and Borgwardt, Karsten M.},
title = {The Evaluation of {T}ools {U}sed to {P}redict the {I}mpact of {M}issense {V}ariants {I}s {H}indered by {T}wo {T}ypes of {C}ircularity},
journal = {Human Mutation},
volume = {36},
number = {5},
issn = {1098-1004},
url = {},
doi = "10.1002/humu.22768",
pages = {513--523},
keywords = {pathogenicity prediction tools, exome sequencing},
year = {2015},

Contact Dominik Grimm for questions.

Page URL:
Sat Jun 24 10:44:38 CEST 2017
© 2017 Eidgenössische Technische Hochschule Zürich