Standard agreement measures for interannota-tor reliability are neither necessary nor sufficient to ensure a high quality corpus. In a case study of word sense annotation, conventional methods for evaluating labels from trained an-notators are contrasted with a probabilistic annotation model applied to crowdsourced data. The annotation model provides far more information , including a certainty measure for each gold standard label; the crowdsourced data was collected at less than half the cost of the conventional approach.