Evaluating combinations of ranked lists and visualizations of inter-document similarity

James Allan, Anton Leuski, Russell Swan, Donald Byrd
2001 Information Processing & Management  
We are interested in how ideas from document clustering can be used to improve the retrieval accuracy of ranked lists in interactive systems. In particular, we are interested in ways to evaluate the eectiveness of such systems to decide how they might best be constructed. In this study, we construct and evaluate systems that present the user with ranked lists and a visualization of inter-document similarities. We ®rst carry out a user study to evaluate the clustering/ranked list combination on
more » ... nstance-oriented retrieval, the task of the TREC-6 Interactive Track. We ®nd that although users generally prefer the combination, they are not able to use it to improve eectiveness. In the second half of this study, we develop and evaluate an approach that more directly combines the ranked list with information from inter-document similarities. Using the TREC collections and relevance judgments, we show that it is possible to realize substantial improvements in eectiveness by doing so, and that although users can use the combined information eectively, the system can provide hints that substantially improve on the user's solo eort. The resulting approach shares much in common with an interactive application of incremental relevance feedback. Throughout this study, we illustrate our work using two prototype systems constructed for these evaluations. The ®rst, AspInQuery, is a classic information retrieval system augmented with a specialized tool for recording information about instances of relevance. The other system, Lighthouse, is a Web-based application that combines a ranked list with a portrayal of inter-document similarity. Lighthouse can work with collections such as TREC, as well as the results of Web search engines. Ó
doi:10.1016/s0306-4573(00)00056-x fatcat:t2kijc2c4za2hkr7xp4sbzzlw4