28,096 Hits in 5.3 sec

Cumulated gain-based evaluation of IR techniques

Kalervo Järvelin, Jaana Kekäläinen
2002 ACM Transactions on Information Systems  
The third one computes the relative-to-the-ideal performance of IR techniques, based on the cumulative gain they are able to yield.  ...  The graphs based on the measures also provide insight into the performance IR techniques and allow interpretation, e.g., from the user point of view.  ...  ACKNOWLEDGEMENTS We thank the FIRE group at University of Tampere for helpful comments, and the IR Lab for programming.  ... 
doi:10.1145/582415.582418 fatcat:jheqo3rghbenxd75q42ztuuqli

Binary and graded relevance in IR evaluations—Comparison of the effects on ranking of IR systems

Jaana Kekäläinen
2005 Information Processing & Management  
In this study the rankings of IR systems based on binary and graded relevance in TREC 7 and 8 data are compared.  ...  Binary precision, and cumulated gain, discounted cumulated gain and normalised discounted cumulated gain are the measures compared.  ...  Acknowledgment This study was funded by Academy of Finland under the grant number 52894.  ... 
doi:10.1016/j.ipm.2005.01.004 fatcat:brackldjarftfiijilvgnrbsti

Cumulated Relative Position: A Metric for Ranking Evaluation [chapter]

Marco Angelini, Nicola Ferro, Kalervo Järvelin, Heikki Keskustalo, Ari Pirkola, Giuseppe Santucci, Gianmaria Silvello
2012 Lecture Notes in Computer Science  
IR research offers a strong evaluation methodology and a range of evaluation metrics, such as MAP and (n)DCG. In this paper, we propose a new metric for ranking evaluation, the CRP.  ...  By cumulating these relative rankings we indicate, at each ranked position, the net effect of document displacements, the CRP.  ...  Acknowledgements The work reported in this paper has been supported by the PROMISE network of excellence (contract n. 258191) project as a part of the 7th Framework Program of the European commission (  ... 
doi:10.1007/978-3-642-33247-0_13 fatcat:cmt2sepjavgsdb7somd2sgfyly

Intuition-supporting visualization of user's performance based on explicit negative higher-order relevance

Heikki Keskustalo, Kalervo Järvelin, Ari Pirkola, Jaana Kekäläinen
2008 Proceedings of the 31st annual international ACM SIGIR conference on Research and development in information retrieval - SIGIR '08  
Modeling the beyond-topical aspects of relevance are currently gaining popularity in IR evaluation.  ...  In this paper, we focus on the concept of negative higher-order relevance (NHOR) made explicit via negative gain values in IR evaluation.  ...  If only the conventional nonnegative gain values are allowed, the normalized values are in the range [0, 1] and they express the share of ideal performance cumulated [4] by the technique evaluated.  ... 
doi:10.1145/1390334.1390448 dblp:conf/sigir/KeskustaloJPK08 fatcat:atra6sikd5ftbmrmyho22xvrqu

Performance Assessment of AMRR and ADCG Metrics in MLIR and IR Systems

Raju Korra, Pothula Sujatha, Author P.Dhavachelvan, Madarapu Naresh Kumar, Sidige Chetana
2011 International Journal of Computer Applications  
The experimental results are evaluated to analyze and compare the performance of proposed MLIR system metrics, Average Mean Reciprocal Rank (AMRR) and Average Discounted Cumulative Gain (ADCG), in Information  ...  Experimental results show that the performance of AMRR, ADCG in MLIR system has been improved 81.67%, 43.93% over IR system respectively.  ...  gain after that evaluated discounted cumulative gain.  ... 
doi:10.5120/2919-3851 fatcat:praeniu7r5a35ptvfz4sqe3s6q

Test-framework: performance profiling and testing web search engine on non factoid queries

Althaf Ali A, Mahammad Shafi R
2019 Indonesian Journal of Electrical Engineering and Computer Science  
of arts approaches in terms of set based measures like Precision, Recall and F measure and rank based measures like Mean Average Precision and Cumulative Gain.  ...  This stands difficult to evaluate the retrieval method in order to schedule or optimize the Recommendation and prediction technique of the IR method to the Real time queries.  ...  engine evaluation against mean average precision 1380 Figure 6 . 6 Performance analysis of search engine evaluation against the cumulative gain The Indonesian J Elec Eng & Comp SciTest-retrieval  ... 
doi:10.11591/ijeecs.v14.i3.pp1373-1381 fatcat:42kxp6frazfmnltvir5nq36awu

A Framework for the Evaluation of Adaptive IR Systems through Implicit Recommendation [chapter]

Catherine Mulwa, Seamus Lawless, M. Rami Ghorab, Eileen O'Donnell, Mary Sharp, Vincent Wade
2011 Lecture Notes in Computer Science  
A Review of evaluation techniques for personalised IR systems was conducted and the results of the analysed survey are presented.  ...  This paper presents a web-based adaptive framework for evaluating personalised information retrieval systems.  ...  Acknowledgements This research is based upon works supported by Science Foundation Ireland (Grant Number: 07/CE/I1142) as part of the Centre for Next Generation Localization (  ... 
doi:10.1007/978-3-642-22688-5_32 fatcat:i5r67jst5fek7op6xm67khlqlu

Visual Comparison of Ranked Result Cumulated Gains [article]

N. Ferro, A. Sabetta, G. Santucci, G. Tino, F. Veltri
2011 International EuroVis Workshop on Visual Analytics  
the cumulative gain up to a given ranked position and taking into account multiple relevance levels.  ...  Ranking is fundamental in Information Retrieval (IR) and several measures have been developed over the years to assess the quality of a ranked result list, such as those based on the idea of computing  ...  Finally, the idea itself of exploring and applying VA techniques to the experimental evaluation in the IR field is quite innovative since it has never been attempted before and, due to the complexity of  ... 
doi:10.2312/pe/eurovast/eurova11/021-024 dblp:conf/eurova-ws/FerroSSTV11 fatcat:k3yuvzgfeze37fb2fyklikcfgu

Evaluating information retrieval system performance based on user preference

Bing Zhou, Yiyu Yao
2009 Journal of Intelligent Information Systems  
The main objective of this paper is to propose a framework for system evaluation based on user preference of documents.  ...  This calls for choosing the proper methods to evaluate the system performance.  ...  We find that most evaluation methods are based on cumulated gain.  ... 
doi:10.1007/s10844-009-0096-5 fatcat:i5qrpg3acnaq3ln3fspl7svsvq

eXtended cumulated gain measures for the evaluation of content-oriented XML retrieval

Gabriella Kazai, Mounia Lalmas
2006 ACM Transactions on Information Systems  
We propose and evaluate a family of metrics, named the eXtended Cumulated Gain (XCG) metrics, for the evaluation of content-oriented XML retrieval approaches.  ...  We evaluate the proposed XCG metrics, based on the INEX test collection, both with respect to their fidelity and reliability.  ...  ACKNOWLEDGEMENTS We are thankful for the help of Paul Ogilvie and Saadia Malik on some of the scripts for the R statistical package.  ... 
doi:10.1145/1185877.1185883 fatcat:lj6dxirjtncn5fpu6xtuamk3ru

Visual Analytics for Information Retrieval Evaluation (VAIRË 2015) [chapter]

Marco Angelini, Nicola Ferro, Giuseppe Santucci, Gianmaria Silvello
2015 Lecture Notes in Computer Science  
The tutorial introduced basic and intermediate concepts about lab-based evaluation of information retrieval systems, its pitfalls, and shortcomings and it complemented them with a recent and innovative  ...  Retrieval (IR) system behaviour.  ...  Some examples of well-known measures, such as Average Precision (AP), Normalized Discounted Cumulated Gain (nDCG), Rank-Biased Precision (RBP) and so on, have been discussed [4] .  ... 
doi:10.1007/978-3-319-16354-3_91 fatcat:aqa3qqai5va55nirrj5bwkkfe4

Beyond classical measures

Azzah Al-Maskari
2007 Proceedings of the 30th annual international ACM SIGIR conference on Research and development in information retrieval - SIGIR '07  
Previous studies have concluded that improvements in IR systems based on increase in IR effectiveness measures do not reflect on improvement in users' performance.  ...  Thus, by the end of this research, we hope to develop a valid and reliable metrics for IIR evaluation.  ...  Thus, we set up a second experiment to investigate how users' satisfaction correlate with some IR effectiveness measures such as precision and the suite of Cumulative Gain measures (CG, DCG, NDCG) in simple  ... 
doi:10.1145/1277741.1277984 dblp:conf/sigir/Al-Maskari07 fatcat:mfxegxtsevafvgzahrchjica4e

Improving Ranking Evaluation Employing Visual Analytics [chapter]

Marco Angelini, Nicola Ferro, Giuseppe Santucci, Gianmaria Silvello
2013 Lecture Notes in Computer Science  
This paper provides an analytical model for examining rankings produced by IR systems, based on the discounted cumulative gain family of metrics, and visualization for performing failure and "what-if"  ...  The work reported in this paper has been supported by the PROMISE network of excellence (contract n. 258191) project as a part of the 7th Framework Program of the European commission (FP7/2007-2013).  ...  Related Work The graded-relevance metrics considered in this paper are based on cumulative gain [5] ; the Discounted Cumulated Gain (DCG) measures are based on the idea that documents are divided in multiple  ... 
doi:10.1007/978-3-642-40802-1_4 fatcat:63vqdlia2rbi7cyjyugk5jj2my

Performance evaluation of non-Gaussian channel estimation techniques on ultrawideband impulse radio channels

Richard J. Barton, Manoj Daniel
2007 International Journal of Communication Systems  
Modern UWB-IR technology is based on transmission of ultra-short pulses with very broad but low-amplitude power spectral densities (PSDs) [1, 2] .  ...  The purpose of the work is to demonstrate that the statistical characteristics of the UWB-IR environment are strongly non-Gaussian and that as a result, non-Gaussian signal processing techniques are both  ...  In this section, we evaluate the performance of the three training-sequence-based channel estimation algorithms.  ... 
doi:10.1002/dac.861 fatcat:yhhpr4jr4beozmotcejlvnjvhm

IR Research: Systems, Interaction, Evaluation and Theories [chapter]

Kalervo Järvelin
2011 Lecture Notes in Computer Science  
We will discuss nested IR evaluation frameworks ranging from IR system centered evaluation to work-task based evaluation.  ...  Such studies do not solve a technical problem, nor evaluate any particular technique, and may therefore be considered unpractical.  ...  Paper Award *) for session-based IR evaluation; the IIiX 2010 Best Paper Award *) for a study on task-based information access; and the Tony Kent Strix Award 2008 in recognition of contributions to the  ... 
doi:10.1007/978-3-642-20161-5_1 fatcat:ndk75j2lrrad7l76awq6qqpb2m
« Previous Showing results 1 — 15 out of 28,096 results