Relevance judgments between TREC and Non-TREC assessors

Azzah Al-Maskari, Mark Sanderson, Paul Clough
2008 Proceedings of the 31st annual international ACM SIGIR conference on Research and development in information retrieval - SIGIR '08  
This paper investigates the agreement of relevance assessments between official TREC judgments and those generated from an interactive IR experiment. Results show that 63% of documents judged relevant by our users matched official TREC judgments. Several factors contributed to differences in the agreements: the number of retrieved relevant documents; the number of relevant documents judged; system effectiveness per topic and the ranking of relevant documents.
doi:10.1145/1390334.1390450 dblp:conf/sigir/Al-MaskariSC08 fatcat:fz26ofjcvbfnfp3fnfs6j5i5py