System effectiveness, user models, and user utility

Ben Carterette
2011 Proceedings of the 34th international ACM SIGIR conference on Research and development in Information - SIGIR '11  
There is great interest in producing effectiveness measures that model user behavior in order to better model the utility of a system to its users. These measures are often formulated as a sum over the product of a discount function of ranks and a gain function mapping relevance assessments to numeric utility values. We develop a conceptual framework for analyzing such effectiveness measures based on classifying members of this broad family of measures into four distinct families, each of which
more » ... reflects a different notion of system utility. Within this framework we can hypothesize about the properties that such a measure should have and test those hypotheses against user and system data. Along the way we present a collection of novel results about specific measures and relationships between them. MODELS We argue that model-based measures are actually composed from three distinct underlying models: 1. a browsing model that describes how a user interacts with results;
doi:10.1145/2009916.2010037 dblp:conf/sigir/Carterette11 fatcat:njyximtvwzc5bg5uhigeximdca