Overview of the ImageCLEF 2006 Photographic Retrieval and Object Annotation Tasks [chapter]

Paul Clough, Michael Grubinger, Thomas Deselaers, Allan Hanbury, Henning Müller
2007 Lecture Notes in Computer Science  
This paper describes the photographic retrieval and object annotation tasks of ImageCLEF 2006. These tasks provide both the resources and the framework necessary to perform comparative laboratorystyle evaluation of visual information systems for image retrieval and automatic image annotation. Both tasks attracted several submissions: 12 groups participating in ImageCLEFphoto and 3 in the automatic annotation task. This paper summarises components of the benchmark, collections, submissions, and
more » ... esults. The photographic retrieval task, ImageCLEFphoto, used a new collection -the IAPR-TC12 Benchmark -of 20,000 colour photographs with semi-structured captions in English and German. For ImageCLEFphoto groups submitted mainly textual runs. However, 31% of runs involved a visual retrieval technique, typically combined with text through the merging of image and text retrieval results. Bilingual text retrieval was performed using two target languages: English and German, with 59% of runs bilingual. Highest monolingual of English was shown to be 74% for Portuguese-English and 39% of German for English-German. Combined text and retrieval approaches were seen to give, on average, higher retrieval results (+54%) than using text (or image) retrieval alone. Similar to previous years, the use of relevance feedback to enable query expansion was seen to improve the text-based submissions by an average of 39%. Topics have been categorised and analysed with respect to attributes including an estimation of their "visualness" and linguistic complexity. The automatic object annotation task used a hand-collected dataset of 81'211 images from 268 classes provided by LTUtech. Given training data, participants were required to classify previously unseen images. The error rate of submissions for this task was high (ranging from 77.3% to 93.2%) resulting in a large proportion of test images being misclassified by any of the proposed classification methods. The task was very challenging for participants.
doi:10.1007/978-3-540-74999-8_71 fatcat:nwavr7byzbbflp3b7wvxd4lem4