Sequence Effects in Crowdsourced Annotations

Nitika Mathur, Timothy Baldwin, Trevor Cohn
2017 Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing  
Manual data annotation is a vital component of NLP research. When designing annotation tasks, properties of the annotation interface can lead to unintentional artefacts in the resulting dataset, biasing the evaluation. In this paper, we explore sequence effects where annotations of an item are affected by the preceding items. Having assigned one label to an instance, the annotator may be less (or more) likely to assign the same label to the next. During rating tasks, seeing a low quality item
more » ... low quality item may affect the score given to the next item either positively or negatively. We see clear evidence of both types of effects using auto-correlation studies over three different crowdsourced datasets. We then recommend a simple way to minimise sequence effects.
doi:10.18653/v1/d17-1306 dblp:conf/emnlp/MathurBC17 fatcat:2oqafxinhzd55m54dstfxkufva