Filters








804 Hits in 4.5 sec

ROC and AUC with a Binary Predictor: a Potentially Misleading Metric [article]

John Muschelli
2019 arXiv   pre-print
Overall, we believe a linear interpolation from the ROC curve with binary predictors, which is most commonly done in software, corresponding to the estimated AUC.  ...  We believe these ROC curves and AUC can lead to misleading results. We compare R, Python, Stata, and SAS software implementations.  ...  Although many discuss the properties of ROC and AUC analyses, we wish to first show the math and calculations of the AUC with a binary predictor and we then explore commonly-used statistical software for  ... 
arXiv:1903.04881v1 fatcat:x3cei2qgffhebavyi3mxe2whle

ROC plot and AUC with binary classifiers: pragmatic analysis of cognitive screening instruments [article]

Gashirai K Mbizvo, Andrew J Larner
2021 medRxiv   pre-print
It has been suggested that ROC and AUC may be potentially misleading when examining binary predictors rather than continuous scales.  ...  The purpose of this study was to examine ROC plots and AUC values for two binary classifiers of cognitive status (applause sign, attended with sign), a cognitive screening instrument producing categorical  ...  Hence the use of AUC is a potentially misleading metric in .  ... 
doi:10.1101/2021.03.09.21253194 fatcat:2ugpx4mwjraavadi4kbq4poqdy

Random Forests, Decision Trees, and Categorical Predictors: The "Absent Levels" Problem [article]

Timothy C. Au
2018 arXiv   pre-print
Furthermore, by using three real data examples, we illustrate how absent levels can dramatically alter a model's performance in practice, and we empirically demonstrate how some simple heuristics can be  ...  Although these incidents may appear to be innocuous, by using Leo Breiman and Adele Cutler's random forests FORTRAN code and the randomForest R package (Liaw and Wiener, 2002) as motivating case studies  ...  The author would also like to thank David Chan, Robert Bell, the action editor, and the anonymous reviewers for their helpful feedback.  ... 
arXiv:1706.03492v2 fatcat:3fcncob2xvdsnncnk2wer3rxmi

Attention to issues and facts

Chris J. Vargo, Tobias Hopp
2019 The Agenda Setting Journal  
A new method is introduced that combines survey data with users' Facebook accounts and their actual Facebook posts to reveal the historical news sharing behaviors of 741 U.S. citizens.  ...  Results suggest that a key component found in need for orientation – attention to relevant issues and facts – predicts observed political news sharing on Facebook.  ...  My understanding of ROC analysis has benefited greatly from discussions with him.  ... 
doi:10.1075/asj.18004.var fatcat:lf64je33uzhj5bsiy2j53n3j3q

Comparison of missing data handling methods for variant pathogenicity predictors [article]

Mikko Ilmari Särkkä, Sami Myöhänen, Kaloyan Marinov, Inka Saarinen, Leo Lahti, Vittorio Fortino, Jussi Paananen
2022 bioRxiv   pre-print
These solutions rely on the comprehensiveness of the available data and struggle with the sparse nature of genetic variant data.  ...  Various machine learning (ML), artificial intelligence (AI), and in silico variant pathogenicity predictors have been developed to solve this challenge.  ...  We also present results with the area under ROC curve (AUC-ROC, or just AUC) metric, defined as the area under the receiver operating characteristic curve.  ... 
doi:10.1101/2022.06.17.496578 fatcat:cgkyrxs6cneydoc5fa5xviok64

ROC curves for clinical prediction models part 1: ROC plots showed no added value above the AUC when evaluating the performance of clinical prediction models

Jan Y. Verbakel, Ewout W. Steyerberg, Hajime Uno, Bavo De Cock, Laure Wynants, Gary S. Collins, Ben Van Calster
2020 Journal of Clinical Epidemiology  
Classification plots can be supplemented with measures such as net benefit to assess the potential clinical utility of a model.  ...  Receiver operating characteristic (ROC) curves are widely used in reports on clinical risk prediction models to demonstrate the ability of a model to discriminate between patients with and without a certain  ...  Acknowledgments We acknowledge English language editing by Dr Jennifer A de Beyer of the Center for Statistics in Medicine, University of Oxford.  ... 
doi:10.1016/j.jclinepi.2020.01.028 pmid:32712176 fatcat:etp3eseudjd3xnq2qupblnhssy

Calibrating variant-scoring methods for clinical decision making

Silvia Benevenuta, Emidio Capriotti, Piero Fariselli
2021 Bioinformatics  
Poorly calibrated algorithms can be misleading and potentially harmful for clinical decision-making. Supplementary information Supplementary data are available at Bioinformatics online.  ...  Calibration refers to the idea that if a model predicts a group of variants to be pathogenic with a probability P, it is expected that the same fraction P of true positive is found in the observed set.  ...  AUC performance of the different predictors One of the most commonly used metrics for assessing the performance of the classification methods is the AUC-ROC ( An ideal classifier would have an AUC-ROC  ... 
doi:10.1093/bioinformatics/btaa943 pmid:33492342 pmcid:PMC8023678 fatcat:ynitweouergs5li46hecnssoxi

Large-scale comparative assessment of computational predictors for lysine post-translational modification sites

2018 Briefings in Bioinformatics  
With this goal in mind, we first provide a comprehensive survey on a large collection of 49 state-of-the-art approaches for lysine PTM prediction.  ...  We cover a variety of important aspects that are crucial for the development of successful predictors, including operating algorithms, sequence and structural features, feature selection, model performance  ...  ROC curves and AUC values of MUscADEL and other predictors for (A) acetylation, (B) glycation, (C) malonylation, (D) methylation, (E) succinylation, (F) sumoylation and (G) ubiquitination.  ... 
doi:10.1093/bib/bby089 pmid:30285084 pmcid:PMC6954452 fatcat:ipqhpnlhufegnodxthnwl75t4y

Binary classifier metrics for optimizing HEP event selection

Andrea Valassi, A. Forti, L. Betev, M. Litmaath, O. Smirnova, P. Hristov
2019 EPJ Web of Conferences  
I discuss the choice of evaluation metrics for binary classifiers in High Energy Physics (HEP) event selection and I point out that the Area Under the ROC Curve (AUC) is of limited relevance in this context  ...  I propose new metrics based on Fisher information, which can be used for both the evaluation and training of HEP event selection algorithms in statistically limited measurements of a parameter.  ...  The main problem with the AUC, both in medical diagnostics and in concrete applications of ML techniques in banking and commerce, is that it is misleading when comparing two ROCs that cross: the AUC is  ... 
doi:10.1051/epjconf/201921406004 fatcat:p5lrs6jiirco7j6j6e2lmvsqra

Multi-temporal analysis reveals that predictors of mountain pine beetle infestation change during outbreak cycles

Jonathan A. Walter, Rutherford V. Platt
2013 Forest Ecology and Management  
In early years, red attack was associated with forest and topographic chara cteristics known to influence susceptibility to MPB.  ...  algorithm with manually selected training classes.  ...  Negrón and an anonymous reviewer for comments that improved the manuscript.  ... 
doi:10.1016/j.foreco.2013.03.038 fatcat:n4ga66m2vnhmbpcmdehq2zo2xi

The importance of prediction model validation and assessment in obesity and nutrition research

A E Ivanescu, P Li, B George, A W Brown, S W Keith, D Raju, D B Allison
2015 International Journal of Obesity  
In this overview, we explain why reductions in predictive validity occur, define the metrics commonly used to estimate the predictive validity of a model (for example, coefficient of determination (R 2  ...  , and adjusted and shrunken R 2 ).  ...  ACKNOWLEDGEMENTS This study was supported in part by NIH grants R25DK099080, R25HL124208 and P30DK056336.  ... 
doi:10.1038/ijo.2015.214 pmid:26449421 pmcid:PMC4826636 fatcat:fjzfccp2yfgltkqjeshpb3aq3u

Partner‐specific Prediction of RNA‐binding Residues in Proteins: A Critical Assessment

Yong Jung, Yasser EL‐Manzalawy, Drena Dobbs, Vasant Honavar
2018 Proteins: Structure, Function, and Bioinformatics  
Specifically, we introduce a novel metric, RNA-specificity metric (RSM), for quantifying the RNA-specificity of the RNA binding residues predicted by such tools.  ...  In this work, we assess the performance of two recently published partner-specific protein-RNA interface prediction tools, PS-PRIP, and PRIdictor, along with our own new tools.  ...  , AUC [ROC], and AUC [CROC], with differences in any metric (if any) at most 2%.  ... 
doi:10.1002/prot.25639 pmid:30536635 pmcid:PMC6389706 fatcat:t22qizjhzvblxdtkz6xd5iwcai

Caveats and pitfalls of ROC analysis in clinical microarray research (and how to avoid them)

D. Berrar, P. Flach
2011 Briefings in Bioinformatics  
ROC curves are frequently summarized in a single scalar, the area under the curve (AUC).  ...  This article discusses the caveats and pitfalls of ROC analysis in clinical microarray research, particularly in relation to (i) the interpretation of AUC (especially a value close to 0.5); (ii) model  ...  To avoid potentially misleading interpretations, it is important to acknowledge the fundamental difference between classification and ranking. ROC measures the latter.  ... 
doi:10.1093/bib/bbr008 pmid:21422066 fatcat:oki4d65ounhxrburkogfdclaga

Who has a Choice?: Survey-Based Predictors of Volitionality in Facebook Use and Non-use

Patrick Skeba, Devansh Saxena, Shion Guha, Eric P. S. Baumer
2021 Proceedings of the ACM on Human-Computer Interaction  
Furthermore, they provide potential implications both for future work and for technology policy.  ...  , a sense of their own agency, and, across both studies, level of education.  ...  IIS-1421498 and Grant No. CNS-1814533) and the Facebook Computational Social Science Methodology Research Award. Thanks to our survey participants for sharing their data and their experiences.  ... 
doi:10.1145/3463935 fatcat:sdhfmr5yzvhtrpdsq6oc6qddcm

Benchmarking machine learning models for the analysis of genetic data using FRESA.CAD Binary Classification Benchmarking [article]

Javier De Velasco Oriol, Antonio Martinez-Torteya, Victor Treviño, Israel Alanis, Edgar E. Vallejo, Jose Gerardo Tamez-Peña
2019 bioRxiv   pre-print
However, with the availability of a wide variety of such methods, model selection has become increasingly difficult, both from the human and computational perspective.  ...  FRESA.CAD Binary Benchmarking demonstrates to be a useful tool over a variety of binary classification problems comprising the analysis of genetic data showing both quantitative and qualitative advantages  ...  the Bioinformatics for Clinical Diagnosis research program, Escuela de Medicina y Ciencias de la Salud, Tecnologico de Monterrey for their valuable comments during the coding of the FRESA.CAD package and  ... 
doi:10.1101/733675 fatcat:gyfatv5etrhermewf6goy7uqsa
« Previous Showing results 1 — 15 out of 804 results