Overview of the ImageCLEF 2012 Scalable Web Image Annotation Task
Conference and Labs of the Evaluation Forum
The ImageCLEF 2012 Scalable Image Annotation Using General Web Data Task proposed a challenge, in which as training data instead of relying only on a set of manually annotated images, the objective was to make use of automatically gathered Web data, with the aim of developing more scalable image annotation systems. To this end, the participants were provided with a new dataset, composed of 250,000 images for training, which included various visual feature types, and textual features obtained
... m the websites in which the images appeared. Two subtasks were defined. The first subtask employed the same test set as the ImageCLEF 2012 Flickr Photo Annotation subtask, with the particularity that both the Flickr and Web training sets had to be used. The idea was to determine if the Web data could help to enhance the annotation performance in comparison to using only manually annotated data. The second subtask consisted in using only automatically gathered Web data to develop an image annotation system. For this, we provided a development and test sets of 1,000 and 2,000 images, respectively, both manually annotated for 95 and 105 concepts, respectively. The participants of the first subtask were not able to take advantage of the Web data to enhance the annotation performance. On the contrary, in the second subtask interesting results were obtained. As expected, the overall performance of the systems is worse than using manually annotated data, nonetheless, the results are promising when analyzing per concept. For some concepts the performance is relatively good, confirming that the Web data can in fact be quite useful. Moreover, due to the low participation and the relatively simple techniques used, it is believed that there is considerable room for improvement on both subtasks.