Using statistical testing in the evaluation of retrieval experiments

David Hull
1993 Proceedings of the 16th annual international ACM SIGIR conference on Research and development in information retrieval - SIGIR '93  
The standard strategies for evaluation based on precision and recall are examined and their relative advantages and disadvantages are discussed. In particular, it is suggested that relevance feedback be evaluated from theperspective of the user. Anumber of different statistical tests are described for determining if differences in performance between retrieval methods are significant. These tests have often been ignored in the past because most are based on an assumption of normality which is
more » ... t strictly valid for the standard performance measures. However, one can test this assumption using simple diagnostic plots, and if it is a poor approximation, there are anumber ofnon-parametric alternatives.
doi:10.1145/160688.160758 dblp:conf/sigir/Hull93 fatcat:4tiviqm5pfbpdpulvcadd67f3q