Information available nowadays in web repositories is big and potentially rich. Natural Language Processing (NLP) systems can play a key role in revealing information: they can analyze and extract relevant information from web content, transforming it into machine-processable annotation data for building knowledge [Campos et al. 2014 ]. Most state-of-the-art NLP systems are largely based on supervised approaches, that is, machine-learning systems that learn how to analyze content, based on

more »

... training texts, referred to as gold standard corpora or, briefly, gold standard. These contain manual annotation data of texts, that is, data manually added by annotation experts. The task of manually annotating a text of a gold standard is hence of paramount importance: since errors in the manual annotation workflow impact the performance of NLP systems, quality control over the annotated data is necessary. Current methods for controlling the quality of annotations are limited: they usually check the local context of the annotation or suggest annotations similar to those already available. A common procedure for keeping annotation data quality under control is the calculation of the so-called interannotator agreement [Artstein and Poesio 2008], a set of statistical measures for computing the agreement of different annotators performing the same task on the same texts: low agreement tends to indicate that the task requires a high degree of personal interpretation or that the annotation guidelines may be poorly written. In spite of existing quality control methods, errors can still be found in