ROC analysis is widely accepted to assess and compare diagnostic validity of laboratory tests. Within the last few years, many new ROC programs have become available but have not been systematically evaluated. The aim of this study was to assess different ROC programs regarding their ease of use, mathematical correctness, final output, and their compatibility with other graphics programs. Methods: Eight available programs running under Windows (AccuROC, Analyse-It, CMDT, GraphROC, Med-Calc,

more »

... , ROCKIT, and SPSS) were evaluated. ROC analyses of prostate-specific antigen and related values were performed from a dataset of 928 men with prostate cancer and benign prostatic hyperplasia and corresponding subsets. Criteria such as data input, data output, and correctness and completeness of results were used to evaluate the practicability of the programs. Results: Although the programs produced equivalent results (areas under the curves and their characteristics), we observed deficiencies concerning input of data, processing of the output data, and completeness of the results. Analyse-It, AccuROC, and MedCalc exhibited good performance, but each program had different shortcomings. Only GraphROC could compare curves at a certain sensitivity or specificity cutoff. Conclusions: Adequate ROC analysis and ROC plotting cannot be performed with a single program. Analyse-It, AccuROC, and MedCalc can be recommended with certain limitations. Further improvements of the programs are necessary. ROC analysis is now a standard tool to assess, define, and compare the diagnostic validity of laboratory tests or diagnostic measures (1 ). Medline searches have shown that the number of publications using ROC curves has increased from ϳ300 studies in the 1980s to Ͼ5000 studies since 1990. Several computer programs have been developed to generate ROC curves, and some of the early programs were briefly described in 1993 (2 ). However, all of these early programs had limitations for easy and accessible practical use. Within the last several years, commercial and public domain programs have become available for complex ROC analysis and ROC plotting. To our knowledge, an overview and comparison of these newly available ROC programs has not been performed. The aims of this study were (a) to survey currently available ROC programs, (b) to compare these ROC programs for their ease of use, and (c) to evaluate their relative utility in ROC analysis.