Internet Archive Scholar

Using permutations instead of student's t distribution for p-values in paired-difference algorithm comparisons

J. Menke, T.R. Martinez
2004 IEEE International Joint Conference on Neural Networks (IEEE Cat. No.04CH37541)  

Preserved Fulltext

fulltext thumbnail
Web Archive Capture PDF (64.3 kB)
https://web.archive.org/web/20070417220952/http://axon.cs.byu.edu/papers/menke_2004_permutations.pdf

A copy of this work was available on the public web and has been preserved in the Wayback Machine. The capture dates from 2007; you can also visit the original URL. The file type is application/pdf.

The paired-difference t-test is commonly used in the machine learning community to determine whether one learning algorithm is better than another on a given learning task. This paper suggests the use of the permutation test instead because it calculates the exact p-value instead of an estimate. The permutation test is also distribution free and the time complexity is trivial for the commonly used 10-fold cross-validation paireddifference test. Results of experiments on real-world problems
more » ... st it is not uncommon to see the t-test estimate deviate up to 30-50% from the exact p-value.
doi:10.1109/ijcnn.2004.1380138 fatcat:bpdyhc5usvetvjlmxavwnkbknq
Citation

J. Menke, T.R. Martinez. "Using permutations instead of student's t distribution for p-values in paired-difference algorithm comparisons." 2004 IEEE International Joint Conference on Neural Networks (IEEE Cat. No.04CH37541)