Semantic concept-based query expansion and re-ranking for multimedia retrieval

Apostol (Paul) Natsev, Alexander Haubold, Jelena Tešić, Lexing Xie, Rong Yan
2007 Proceedings of the 15th international conference on Multimedia - MULTIMEDIA '07  
We study the problem of semantic concept-based query expansion and re-ranking for multimedia retrieval. In particular, we explore the utility of a fixed lexicon of visual semantic concepts for automatic multimedia retrieval and re-ranking purposes. In this paper, we propose several new approaches for query expansion, in which textual keywords, visual examples, or initial retrieval results are analyzed to identify the most relevant visual concepts for the given query. These concepts are then
more » ... to generate additional query results and/or to re-rank an existing set of results. We develop both lexical and statistical approaches for text query expansion, as well as content-based approaches for visual query expansion. In addition, we study several other recently proposed methods for concept-based query expansion. In total, we compare 7 different approaches for expanding queries with visual semantic concepts. They are evaluated using a large video corpus and 39 concept detectors from the TRECVID-2006 video retrieval benchmark. We observe consistent improvement over the baselines for all 7 approaches, leading to an overall performance gain of 78% relative to a text retrieval baseline, and a 31% improvement relative to a stateof-the-art multimodal retrieval baseline. To the best of our knowledge, this is the most comprehensive review and evaluation of concept-based retrieval and re-ranking methods so far, and it clearly establishes the value of semantic concept detectors for answering and expanding ad-hoc user queries. Figure 1 : Overview of concept-based retrieval and re-ranking framework. Three general approaches are illustrated for identifying relevant semantic concepts to a query-based on textual query analysis, visual content-based query modeling, and pseudorelevance feedback. A multi-modal fusion step leverages the identified relevant concepts to expand and re-rank the results. Concept-based expanded and re-ranked results Multimedia Repository
doi:10.1145/1291233.1291448 dblp:conf/mm/NatsevHTXY07 fatcat:lqjfylum3bgwtkepponnjx6f3i