The evaluation of the quality of ontological classification is an important part of semantic web technology. Because this area is under constant development, it requires improvement and standardisation. This paper discusses existing evaluation metrics, and proposes a new method for evaluating the ontology population task, which is general enough to be used in a variety of situations, yet more precise than many current metrics. The paper further describes our first efforts in operationalising

more »

... evaluation procedure, including the creation of a semantically annotated corpus that will function as a test bed for the proposed evaluation mechanism, and comparison of different evaluation metrics. We conclude that for ontology-based evaluation, a more complex mechanism than is traditionally used is preferable. This mechanism aims to drive a benchmarking assessment tool for the current state-of-the-art of ontology population, and to set a standard for best practice for future evaluation of human language technology for the semantic web.