A unified model for metasearch, pooling, and system evaluation

Javed A. Aslam, Virgiliu Pavlu, Robert Savell
2003 Proceedings of the twelfth international conference on Information and knowledge management - CIKM '03  
We present a unified model which, given the ranked lists of documents returned by multiple retrieval systems in response to a given query, simultaneously solves the problems of (1) fusing the ranked lists of documents in order to obtain a high-quality combined list (metasearch); (2) generating document collections likely to contain large fractions of relevant documents (pooling); and (3) accurately evaluating the underlying retrieval systems with small numbers of relevance judgments (efficient
more » ... ystem assessment). Our approach is based on the Hedge algorithm for on-line learning. In effect, our proposed system "learns" which documents are likely to be relevant from a sequence of on-line relevance judgments. In experiments using TREC data, our methodology is shown to outperform standard methods for metasearch, pooling, and system evaluation, often remarkably so.
doi:10.1145/956950.956953 fatcat:yow6vkikfvasjkvqexiwnxvbwq