Instrumenting the crowd

Jeffrey M. Rzeszotarski, Aniket Kittur
2011 Proceedings of the 24th annual ACM symposium on User interface software and technology - UIST '11  
Detecting and correcting low quality submissions in crowdsourcing tasks is an important challenge. Prior work has primarily focused on worker outcomes or reputation, using approaches such as agreement across workers or with a gold standard to evaluate quality. We propose an alternative and complementary technique that focuses on the way workers work rather than the products they produce. Our technique captures behavioral traces from online crowd workers and uses them to predict outcome measures
more » ... ct outcome measures such quality, errors, and the likelihood of cheating. We evaluate the effectiveness of the approach across three contexts including classification, generation, and comprehension tasks. The results indicate that we can build predictive models of task performance based on behavioral traces alone, and that these models generalize to related tasks. Finally, we discuss limitations and extensions of the approach.
doi:10.1145/2047196.2047199 dblp:conf/uist/RzeszotarskiK11 fatcat:bxng3puo5rhn3lm2zvt3y7qsnq