It. is common in NLP that the categories into which text is classified do not have fully objective definitions. Examples of such categories are lexical distinctions such as part-of-speech tags and wordsense distinctions, sentence level distinctions such as phrase attachment, and discourse level distinct.icms such as topic or speech-act categorization. This p>1per presents an approach to analy?-ing the agrcen1ent arnong lnnnan judges for the purpose of formulating a refined and more reliable set

more »

... of category designations. We use these techniques to analyze the sense tags assigned by five judgps to the noun intcr•est. The initial tag set is takmi from Longman's Dictionary of Contemporary }i:nglish. Through this process of analysis, we automatically identify and assign a revised set of sense tags for the data. The revised tags exhibit high reliability as measured by Cohen's r;.. Such techniques are important for formulating and evaluating both human and automated classification systems.