Effectively Crowdsourcing Radiology Report Annotations

Anne Cocos, Aaron Masino, Ting Qian, Ellie Pavlick, Chris Callison-Burch
2015 Proceedings of the Sixth International Workshop on Health Text Mining and Information Analysis  
Crowdsourcing platforms are a popular choice for researchers to gather text annotations quickly at scale. We investigate whether crowdsourced annotations are useful when the labeling task requires medical domain knowledge. Comparing a sentence classification model trained with expert-annotated sentences to the same model trained on crowd-labeled sentences, we find the crowdsourced training data to be just as effective as the manually produced dataset. We can improve the accuracy of the
more » ... acy of the crowd-fueled model without collecting further labels by filtering out worker labels applied with low confidence.
doi:10.18653/v1/w15-2614 dblp:conf/acl-louhi/CocosMQPC15 fatcat:ntwx7jwfjnfr7ikzmsoviiv6v4