Assessing performance of pathogenicity predictors using clinically relevant variant datasets

Adam C Gunning, Verity Fryer, James Fasham, Andrew H Crosby, Sian Ellard, Emma L Baple, Caroline F Wright
<span title="2020-08-25">2020</span> <i title="BMJ"> <a target="_blank" rel="noopener" href="https://fatcat.wiki/container/h4x5oiufxvdpxb6kfbjrpd6ble" style="color: black;">Journal of Medical Genetics</a> </i> &nbsp;
BackgroundPathogenicity predictors are integral to genomic variant interpretation but, despite their widespread usage, an independent validation of performance using a clinically relevant dataset has not been undertaken.MethodsWe derive two validation datasets: an 'open' dataset containing variants extracted from publicly available databases, similar to those commonly applied in previous benchmarking exercises, and a 'clinically representative' dataset containing variants identified through
more &raquo; ... arch/diagnostic exome and panel sequencing. Using these datasets, we evaluate the performance of three recent meta-predictors, REVEL, GAVIN and ClinPred, and compare their performance against two commonly used in silico tools, SIFT and PolyPhen-2.ResultsAlthough the newer meta-predictors outperform the older tools, the performance of all pathogenicity predictors is substantially lower in the clinically representative dataset. Using our clinically relevant dataset, REVEL performed best with an area under the receiver operating characteristic curve of 0.82. Using a concordance-based approach based on a consensus of multiple tools reduces the performance due to both discordance between tools and false concordance where tools make common misclassification. Analysis of tool feature usage may give an insight into the tool performance and misclassification.ConclusionOur results support the adoption of meta-predictors over traditional in silico tools, but do not support a consensus-based approach as in current practice.
<span class="external-identifiers"> <a target="_blank" rel="external noopener noreferrer" href="https://doi.org/10.1136/jmedgenet-2020-107003">doi:10.1136/jmedgenet-2020-107003</a> <a target="_blank" rel="external noopener" href="https://www.ncbi.nlm.nih.gov/pubmed/32843488">pmid:32843488</a> <a target="_blank" rel="external noopener" href="https://pubmed.ncbi.nlm.nih.gov/PMC8327323/">pmcid:PMC8327323</a> <a target="_blank" rel="external noopener" href="https://fatcat.wiki/release/p5arsgtt7janffoj2lgfl6almu">fatcat:p5arsgtt7janffoj2lgfl6almu</a> </span>
<a target="_blank" rel="noopener" href="https://web.archive.org/web/20200828224113/https://jmg.bmj.com/content/jmedgenet/early/2020/08/25/jmedgenet-2020-107003.full.pdf" title="fulltext PDF download" data-goatcounter-click="serp-fulltext" data-goatcounter-title="serp-fulltext"> <button class="ui simple right pointing dropdown compact black labeled icon button serp-button"> <i class="icon ia-icon"></i> Web Archive [PDF] <div class="menu fulltext-thumbnail"> <img src="https://blobs.fatcat.wiki/thumbnail/pdf/32/a0/32a055391d540b79a8e07b4eb0e55710ff06c40b.180px.jpg" alt="fulltext thumbnail" loading="lazy"> </div> </button> </a> <a target="_blank" rel="external noopener noreferrer" href="https://doi.org/10.1136/jmedgenet-2020-107003"> <button class="ui left aligned compact blue labeled icon button serp-button"> <i class="unlock alternate icon" style="background-color: #fb971f;"></i> bmj.org </button> </a> <a target="_blank" rel="external noopener" href="https://www.ncbi.nlm.nih.gov/pmc/articles/PMC8327323" title="pubmed link"> <button class="ui compact blue labeled icon button serp-button"> <i class="file alternate outline icon"></i> pubmed.gov </button> </a>