A copy of this work was available on the public web and has been preserved in the Wayback Machine. The capture dates from 2019; you can also visit the original URL.
The file type is
On the Relation Between Assessor's Agreement and Accuracy in Gamified Relevance Assessment
Proceedings of the 38th International ACM SIGIR Conference on Research and Development in Information Retrieval - SIGIR '15
Expert judgments (labels) are widely used in Information Retrieval for the purposes of search quality evaluation and machine learning. Setting up the process of collecting such judgments is a challenge of its own, and the maintenance of judgments quality is an extremely important part of the process. One of the possible ways of controlling the quality is monitoring inter-assessor agreement level. But does the agreement level really reflect the quality of assessor's judgments? Indeed, if a groupdoi:10.1145/2766462.2767727 dblp:conf/sigir/MegorskayaKS15 fatcat:vfl33ksuajb7td2xfafjxrnx5i