Estimating interleaved comparison outcomes from historical click data

Katja Hofmann, Shimon Whiteson, Maarten de Rijke
2012 Proceedings of the 21st ACM international conference on Information and knowledge management - CIKM '12  
Interleaved comparison methods, which compare rankers using click data, are a promising alternative to traditional information retrieval evaluation methods that require expensive explicit judgments. A major limitation of these methods is that they assume access to live data, meaning that new data must be collected for every pair of rankers compared. We investigate the use of previously collected click data (i.e., historical data) for interleaved comparisons. We start by analyzing to what degree
more » ... existing interleaved comparison methods can be applied and find that a recent probabilistic method allows such data reuse, even though it is biased when applied to historical data. We then propose an interleaved comparison method that is based on the probabilistic approach but uses importance sampling to compensate for bias. We experimentally confirm that probabilistic methods make the use of historical data for interleaved comparisons possible and effective.
doi:10.1145/2396761.2398516 dblp:conf/cikm/HofmannWR12 fatcat:aaokjqydcbb4zmhcri7bwvjxwy