A copy of this work was available on the public web and has been preserved in the Wayback Machine. The capture dates from 2017; you can also visit the original URL.
The file type is application/pdf
.
Measuring assessor accuracy
2011
Proceedings of the 34th international ACM SIGIR conference on Research and development in Information - SIGIR '11
In many situations, humans judging document relevance are forced to trade-off accuracy for speed. The development of better interactive retrieval systems and relevance assessing platforms requires the measurement of assessor accuracy, but to date the subjective nature of relevance has prevented such measurement. To quantify assessor performance, we define relevance to be a group's majority opinion, and demonstrate the value of this approach by comparing the performance of NIST assessors to a
doi:10.1145/2009916.2010134
dblp:conf/sigir/SmuckerJ11
fatcat:zhkymrw4rffj3nzc2kbbphw454