Using permutations instead of student's t distribution for p-values in paired-difference algorithm comparisons

J. Menke, T.R. Martinez
2004 IEEE International Joint Conference on Neural Networks (IEEE Cat. No.04CH37541)  
The paired-difference t-test is commonly used in the machine learning community to determine whether one learning algorithm is better than another on a given learning task. This paper suggests the use of the permutation test instead because it calculates the exact p-value instead of an estimate. The permutation test is also distribution free and the time complexity is trivial for the commonly used 10-fold cross-validation paireddifference test. Results of experiments on real-world problems
more » ... st it is not uncommon to see the t-test estimate deviate up to 30-50% from the exact p-value.
doi:10.1109/ijcnn.2004.1380138 fatcat:bpdyhc5usvetvjlmxavwnkbknq