Crowdsourcing syntactic relatedness judgements for opinion mining in the study of information technology adoption

Asad B. Sayeed, Bryan Rusk, Martin Petrov, Hieu C. Nguyen, Timothy J. Meyer, Amy Weinberg
2011 Workshop on Language Technology for Cultural Heritage, Social Sciences, and Humanities  
We present an end-to-end pipeline including a user interface for the production of wordlevel annotations for an opinion-mining task in the information technology (IT) domain. Our pre-annotation pipeline selects candidate sentences for annotation using results from a small amount of trained annotation to bias the random selection over a large corpus. Our user interface reduces the need for the user to understand the "meaning" of opinion in our domain context, which is related to community
more » ... n. It acts as a preliminary buffer against low-quality annotators. Finally, our post-annotation pipeline aggregates responses and applies a more aggressive quality filter. We present positive results using two different evaluation philosophies and discuss how our design decisions enabled the collection of high-quality annotations under subjective and fine-grained conditions.
dblp:conf/latech/SayeedRPNMW11 fatcat:fjiaqtdywvaw3a5oreveeyndiu