Ranking the annotators: An agreement study on argumentation structure

Andreas Peldszus, Manfred Stede
2013 Linguistic Annotation Workshop  
We investigate methods for evaluating agreement among a relatively large group of annotators who have not received extensive training and differ in terms of ability and motivation. We show that it is possible to isolate a reliable subgroup of annotators, so that aspects of the difficulty of the underlying task can be studied. Our task is to annotate the argumentative structure of short texts.
dblp:conf/acllaw/PeldszusS13 fatcat:w2v64a4wbngbjisqdvw66goznq