Filters








203 Hits in 5.2 sec

How Do Gain and Discount Functions Affect the Correlation between DCG and User Satisfaction? [chapter]

Julián Urbano, Mónica Marrero
2015 Lecture Notes in Computer Science  
We present an empirical analysis of the effect that the gain and discount functions have in the correlation between DCG and user satisfaction.  ...  We study this relationship for 36 combinations of gain and discount, and find that a linear gain and a constant discount are best correlated with user satisfaction.  ...  Work supported by an A4U postdoctoral grant and the Spanish Government (HAR2011-27540). We thank the reviewers for their comments.  ... 
doi:10.1007/978-3-319-16354-3_20 fatcat:ukkzgmhjdfd57m2vokazuf3syu

Reproduction Study - How do Gain and Discount FunctionsAffect the Correlation between DCG and User Satisfaction?

Maximilian Michel, Alexander Telenkov, Matteo Rossi Reich
2021 Zenodo  
The selected experiment reflects an empirical analysis of the effect that the gain and discount functions have on the correlation between Discounted Cumulative Gain (DCG) and user satisfaction.  ...  The authors come to the conclusion, that a combination of linear gain and constant discount shows the best correlation with user satisfaction.  ...  INTRODUCTION The paper How do Gain and Discount Functions Affect the Correlation between DCG and User Satisfaction?  ... 
doi:10.5281/zenodo.4460547 fatcat:cdo7nu3unfdptjztgtco44uxru

A semi-supervised approach to modeling web search satisfaction

Ahmed Hassan
2012 Proceedings of the 35th international ACM SIGIR conference on Research and development in information retrieval - SIGIR '12  
Web search is an interactive process that involves actions from Web search users and responses from the search engine.  ...  We present a semi-supervised approach to modeling Web search satisfaction. The proposed approach can use either labeled data only or both labeled and unlabeled data.  ...  Huffman and Hochster [15] studied the correlation between user satisfaction and simple relevance metrics.  ... 
doi:10.1145/2348283.2348323 dblp:conf/sigir/Hassan12 fatcat:3afjszmsjncixe5qoflhmcndjq

"Satisfaction with Failure" or "Unsatisfied Success"

Mengyang Liu, Yiqun Liu, Jiaxin Mao, Cheng Luo, Min Zhang, Shaoping Ma
2018 Proceedings of the 2018 World Wide Web Conference on World Wide Web - WWW '18  
In this study, we investigate the differences between user satisfaction and search success, and try to adopt the findings to predict search success in complex search tasks.  ...  The factors (e.g. document readability and credibility) that lead to the inconsistency of search success and user satisfaction are also investigated and adopted to predict whether one search task is successful  ...  [2] found that user satisfaction is strongly correlated with some evaluation metrics such as CG and DCG. Jiang et al.  ... 
doi:10.1145/3178876.3186065 dblp:conf/www/LiuLMLZM18 fatcat:y3wpwnh7wjdu5a33ucixd46nmy

A task level metric for measuring web search satisfaction and its application on improving relevance estimation

Ahmed Hassan, Yang Song, Li-wei He
2011 Proceedings of the 20th ACM international conference on Information and knowledge management - CIKM '11  
We use our user satisfaction model to distinguish between clicks that lead to satisfaction and clicks that do not.  ...  Understanding the behavior of satisfied and unsatisfied Web search users is very important for improving users search experience.  ...  Huffman and Hochster [12] study the correlation between user satisfaction and simple relevance metrics.  ... 
doi:10.1145/2063576.2063599 dblp:conf/cikm/HassanSH11 fatcat:pwvqpinlkjhuzhb3nxb7zosawy

Predicting user preferences

Pavel Sirotkin
2020 Zenodo  
Using this measure, we evaluate the metrics Discounted Cumulated Gain, Mean Average Precision and classical precision, finding that the former performs best.  ...  We propose a new measure for a metric's ability to identify user preference of result lists.  ...  The results showed a strong correlation between average precision and user success metrics (such as the number of retrieved documents) as well as user satisfaction.  ... 
doi:10.5281/zenodo.4133313 fatcat:g3bzpsdm3rhnbgqbrxi3sm227m

Predicting User Preferences [article]

Pavel Sirotkin
2011 arXiv   pre-print
Using this measure, we evaluate the metrics Discounted Cumulated Gain, Mean Average Precision and classical precision, finding that the former performs best.  ...  We propose a new measure for a metric's ability to identify user preference of result lists.  ...  DCG enhances this rather simple method by introducing "[a] discounting function [...]  ... 
arXiv:1103.2886v1 fatcat:lzaitkwihjg5tkvfbhvdrdkjkm

Cumulated gain-based evaluation of IR techniques

Kalervo Järvelin, Jaana Kekäläinen
2002 ACM Transactions on Information Systems  
This article proposes three novel measures that compute the cumulative gain the user obtains by examining the retrieval result up to a given ranked position.  ...  The graphs based on the measures also provide insight into the performance IR techniques and allow interpretation, e.g., from the user point of view.  ...  ACKNOWLEDGEMENTS We thank the FIRE group at University of Tampere for helpful comments, and the IR Lab for programming.  ... 
doi:10.1145/582415.582418 fatcat:jheqo3rghbenxd75q42ztuuqli

From a User Model for Query Sessions to Session Rank Biased Precision (sRBP)

Aldo Lipani, Ben Carterette, Emine Yilmaz
2019 Proceedings of the 2019 ACM SIGIR International Conference on Theory of Information Retrieval - ICTIR '19  
To satisfy their information needs, users usually carry out searches on retrieval systems by continuously trading off between the examination of search results retrieved by under-specified queries and  ...  We demonstrate the quality of this new session-based evaluation measure, named Session RBP (sRBP), by evaluating its user model against the observed user behaviour over the query-sessions of the 2014 TREC  ...  ACKNOWLEDGMENTS This project was funded by the EPSRC Fellowship titled "Task Based Information Retrieval", grant reference number EP/P024289/1.  ... 
doi:10.1145/3341981.3344216 dblp:conf/ictir/LipaniCY19 fatcat:v4i27akdwrdnnia74p3gi462cy

User behavior modeling for Web search evaluation

Fan Zhang, Yiqun Liu, Jiaxin Mao, Min Zhang, Shaoping Ma
2020 AI Open  
From the overview of these metrics, we can see how the assumptions and modeling methods of user behavior have evolved with time.  ...  We also show the methods to compare the performances of model-based evaluation metrics in terms of modeling user behavior and measuring user satisfaction.  ...  between the user model and user satisfaction.  ... 
doi:10.1016/j.aiopen.2021.02.003 fatcat:ir4hopyfy5asdoeddc3y64vk3a

Empirical justification of the gain and discount function for nDCG

Evangelos Kanoulas, Javed A. Aslam
2009 Proceeding of the 18th ACM conference on Information and knowledge management - CIKM '09  
In this work we discuss how to empirically derive a gain and discount function that optimizes the efficiency or stability of nDCG.  ...  First, we describe a variance decomposition analysis framework and an optimization procedure utilized to find the efficiency-or stability-optimal gain and discount functions.  ...  [1] exhibited that cumulative gain without discounting (CG) is more correlated to user satisfaction than discounted cumulative gain (DCG) and nDCG (at least when computed at rank 100).  ... 
doi:10.1145/1645953.1646032 dblp:conf/cikm/KanoulasA09 fatcat:f3tmyjdfmbbq7cvxfaqsi6rml4

Advances on the development of evaluation measures

Ben Carterette, Evangelos Kanoulas, Emine Yilmaz
2012 Proceedings of the 35th international ACM SIGIR conference on Research and development in information retrieval - SIGIR '12  
Use the query logs to understand how users behave• Learn the parameters of the user model from the query logs -Utility, discount, etc.  ...  SIGIR 2011] The user steps down a ranked list of documents until a decision point and either abandons the query or reformulates [Stochastic; allows early abandonment] ••• DCG can be written as:• Discount  ...  • But it has appealing properties other IA measures do not have: ranges between 0 and 1 -submodularity: diminishing returns for relevance to a given subtopic -> built-in redundancy penalization • Also  ... 
doi:10.1145/2348283.2348542 dblp:conf/sigir/CarteretteKY12 fatcat:2ha3q5onmrghpckfnpjl6p6uvm

Evaluating new search engine configurations with pre-existing judgments and clicks

Umut Ozertem, Rosie Jones, Benoit Dumoulin
2011 Proceedings of the 20th international conference on World wide web - WWW '11  
correlated (0.74) with DCG values evaluated using fully judged datasets, and approaches inter-annotator agreement.  ...  Since the user browsing model and the pre-existing editorial data cannot provide relevance estimates for all documents for the selected set of queries, one important challenge is to obtain this performance  ...  Our experiments focus specifically on estimating the difference in Discounted Cumulative Gain (DCG) [9] between two search models (we will refer to this as ∆DCG), since this metric is widely used in  ... 
doi:10.1145/1963405.1963463 dblp:conf/www/OzertemJD11 fatcat:yqab3wair5b7voqxqpqwfcn5ti

Visual interactive failure analysis

Marco Angelini, Nicola Ferro, Gianmaria Silvello, Giuseppe Santucci
2012 Proceedings of the 4th Information Interaction in Context Symposium on - IIIX '12  
This paper provides an analytical model for examining performances of IR systems, based on the discounted cumulative gain family of metrics, and visualization for interacting and exploring the performances  ...  Their development calls for proper evaluation methodologies to ensure that they meet the expected user requirements and provide the desired effectiveness.  ...  Typical instantiations of DCG measures make use of positive gains and logarithmic functions to smooth the discount for higher ranks -e.g. a log 2 function is used to model impatient users while a log 10  ... 
doi:10.1145/2362724.2362757 dblp:conf/iiix/AngeliniFSS12 fatcat:i5i5munun5chtoq2p5nybyl7ka

Cumulated Relative Position: A Metric for Ranking Evaluation [chapter]

Marco Angelini, Nicola Ferro, Kalervo Järvelin, Heikki Keskustalo, Ari Pirkola, Giuseppe Santucci, Gianmaria Silvello
2012 Lecture Notes in Computer Science  
The development of multilingual and multimedia information access systems calls for proper evaluation methodologies to ensure that they meet the expected user requirements and provide the desired effectiveness  ...  IR research offers a strong evaluation methodology and a range of evaluation metrics, such as MAP and (n)DCG. In this paper, we propose a new metric for ranking evaluation, the CRP.  ...  Acknowledgements The work reported in this paper has been supported by the PROMISE network of excellence (contract n. 258191) project as a part of the 7th Framework Program of the European commission (  ... 
doi:10.1007/978-3-642-33247-0_13 fatcat:cmt2sepjavgsdb7somd2sgfyly
« Previous Showing results 1 — 15 out of 203 results