A comparison of statistical significance tests for information retrieval evaluation

Mark D. Smucker, James Allan, Ben Carterette
2007 Proceedings of the sixteenth ACM conference on Conference on information and knowledge management - CIKM '07  
Information retrieval (IR) researchers commonly use three tests of statistical significance: the Student's paired t-test, the Wilcoxon signed rank test, and the sign test. Other researchers have previously proposed using both the bootstrap and Fisher's randomization (permutation) test as nonparametric significance tests for IR but these tests have seen little use. For each of these five tests, we took the ad-hoc retrieval runs submitted to TRECs 3 and 5-8, and for each pair of runs, we measured
more » ... the statistical significance of the difference in their mean average precision. We discovered that there is little practical difference between the randomization, bootstrap, and t tests. Both the Wilcoxon and sign test have a poor ability to detect significance and have the potential to lead to false detections of significance. The Wilcoxon and sign tests are simplified variants of the randomization test and their use should be discontinued for measuring the significance of a difference between means.
doi:10.1145/1321440.1321528 dblp:conf/cikm/SmuckerAC07 fatcat:yysoebqcxvaxjcuegw3mtdetdm