Filters








652,005 Hits in 4.2 sec

On the evaluation of IR systems

S.E. Robertson, M.M. Hancock-Beaulieu
1992 Information Processing & Management  
Published version The volume edited by Sparck Jones and published in 1981, Information Retrieval Experiment, remains the one substantial work on the evaluation of IR systems.  ...  The paper highlights the ever increasing complexity in the evaluation of IR systems which has arisen over the last decade.  ...  Acknowledgement -We would like to thank Donna Harman and a referee for valuable comments on an earlier draft of the paper.  ... 
doi:10.1016/0306-4573(92)90004-j fatcat:qpi33c2y25fyfftdtwe2ggehca

The effect of assessor error on IR system evaluation

Ben Carterette, Ian Soboroff
2010 Proceeding of the 33rd international ACM SIGIR conference on Research and development in information retrieval - SIGIR '10  
Recent efforts in test collection building have focused on scaling back the number of necessary relevance judgments and then scaling up the number of search topics.  ...  We find that while averages are robust, assessor errors can have a large effect on system rankings.  ...  After re-evaluating Million Query track systems, the simplest approach to determining the effect of errors is to measure how well the new evaluation correlates to the "true" evaluation resulting from using  ... 
doi:10.1145/1835449.1835540 dblp:conf/sigir/CarteretteS10 fatcat:grsfjcy34rg6bgpw3mrjs6bygq

Binary and graded relevance in IR evaluations—Comparison of the effects on ranking of IR systems

Jaana Kekäläinen
2005 Information Processing & Management  
In this study the rankings of IR systems based on binary and graded relevance in TREC 7 and 8 data are compared.  ...  Twenty-one topics and 90 systems from TREC 7 and 20 topics and 121 systems from TREC 8 form the data.  ...  Acknowledgment This study was funded by Academy of Finland under the grant number 52894.  ... 
doi:10.1016/j.ipm.2005.01.004 fatcat:brackldjarftfiijilvgnrbsti

The Effect of Inter-Assessor Disagreement on IR System Evaluation: A Case Study with Lancers and Students

Tetsuya Sakai
2017 NTCIR Conference on Evaluation of Information Access Technologies  
We then compared the system rankings and statistical signi cance test results according to di erent qrels versions created by changing which asessors to rely on: overall, the outcomes do di er according  ...  is paper reports on a case study on the inter-assessor disagreements in the English NTCIR-13 We Want Web (WWW) collection.  ...  ACKNOWLEDGEMENTS I thank the PLY team (Peng Xiao, Lingtao Li, Yimeng Fan) of my laboratory for developing the PLY relevance assessment tool and collecting the assessments.  ... 
dblp:conf/ntcir/Sakai17b fatcat:cv6mlyfzjjgsroiikas3t5agfy

Maximum Entropy and the Method of Moments in Performance Evaluation of Digital Communications Systems

M. Kavehrad, M. Joseph
1986 IRE Transactions on Communications Systems  
The method requires about the same number of moments as techniques based on orthogonal expansions.  ...  The maximum entropy criterion for estimating an unknown probability density function from its moments is applied to the evaluation of the average error probability in digital communications.  ...  Mead of the University of Southern Mississippi for many stimulating discussions we had during this work, and for his many helpful comments. Also, many thanks to G. J.  ... 
doi:10.1109/tcom.1986.1096484 fatcat:pgektqgf2fb6dj2kck2gjn3gjq

Continuous Result Delta Evaluation of IR Systems

Gabriela González-Sáez
2022 Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval  
The expected contributions of the thesis are: (i) a pivot strategy based on 𝑅Δ to compare systems evaluated in different EEs; (ii) a formalization of DTC to simulate the continuous evaluation and provide  ...  It is not possible to measure the 𝑅Δ of two systems evaluated in different EEs, because the performance variations are dependent on the changes in the EEs. [1] .  ...  with the absolute performance values for each system evaluated in the different EEs. The correctness of the RoS depends on the system defined as pivot and the metric.  ... 
doi:10.1145/3477495.3531686 fatcat:a3x5kwpaljhgxnimfs4v2fq23m

The TREC-like evaluation of music IR systems

J. Stephen Downie
2003 Proceedings of the 26th annual international ACM SIGIR conference on Research and development in informaion retrieval - SIGIR '03  
The proposed research tasks are based upon expert opinion garnered from members of the Information Retrieval (IR), MDL and MIR communities with regard to the construction and implementation of scientifically  ...  This poster reports upon the ongoing efforts being made to establish TREC-like and other comprehensive evaluation paradigms within the Music IR (MIR) and Music Digital Library (MDL) research communities  ...  How do we integrate the evaluation of MIR systems with the larger framework of IR evaluation (i.e., What aspects are held in common and what are unique to MIR?)? 5.  ... 
doi:10.1145/860435.860547 dblp:conf/sigir/Downie03 fatcat:rkodsdbojjfs3lqedm72ws52ca

Practical evaluation of IR within automated classification systems

R. Dolin, J. Pierre, M. Butler, R. Avedon
1999 Proceedings of the eighth international conference on Information and knowledge management - CIKM '99  
This paper describes some of the work we have done to evaluate and compare the use of three IR systems (Verity, LSI, and SMART) as black boxes within an automated classification environment.  ...  In so doing, we also develop criteria for the construction of a useful training set. These results lead to metrics useful in the integration of IR systems into larger applications.  ...  In this paper, we focus on the development of a methodology for practical evaluation of IR systems, mainly from the perspective of system designers, within the context of an automated classification environment  ... 
doi:10.1145/319950.320023 dblp:conf/cikm/DolinPBA99 fatcat:5ueao72o3nhpphalz3fzwrloxe

The OKPU System in NTCIR11 MedNLP2: An IR Approach to ICD-10 Code Identification

Genichiro Kikui, Yasuhiro Tajima
2014 NTCIR Conference on Evaluation of Information Access Technologies  
Preliminary evaluation for the MedNLP2 test set shows that with this simple approach our system correctly identified 54% of the input medical terms.  ...  This paper describes an IR (Information Retrieval) approach to identifying the ICD-10 code of a medical term, such as a disease name or a description of a symptom or a complaint), in a medical text.  ...  We would also appreciate Professor Shuji Kaneko for allowing us to use the life Science Dictionary.  ... 
dblp:conf/ntcir/KikuiT14 fatcat:kdjrhsbzzbbxniukbl3mnk2rxe

The TREC-like evaluation of music IR systems

J. Stephen Downie
2003 Proceedings of the 26th annual international ACM SIGIR conference on Research and development in informaion retrieval - SIGIR '03  
The proposed research tasks are based upon expert opinion garnered from members of the Information Retrieval (IR), MDL and MIR communities with regard to the construction and implementation of scientifically  ...  This poster reports upon the ongoing efforts being made to establish TREC-like and other comprehensive evaluation paradigms within the Music IR (MIR) and Music Digital Library (MDL) research communities  ...  How do we integrate the evaluation of MIR systems with the larger framework of IR evaluation (i.e., What aspects are held in common and what are unique to MIR?)? 5.  ... 
doi:10.1145/860500.860547 fatcat:gszllm57t5ckvj2emvrzdczevq

User Variability and IR System Evaluation

Peter Bailey, Alistair Moffat, Falk Scholer, Paul Thomas
2015 Proceedings of the 38th International ACM SIGIR Conference on Research and Development in Information Retrieval - SIGIR '15  
We explore two aspects of user variability with regard to evaluating the relative performance of IR systems, assessing effectiveness in the context of a subset of topics from three TREC collections, with  ...  Test collection design eliminates sources of user variability to make statistical comparisons among information retrieval (IR) systems more affordable.  ...  Acknowledgment This work was supported by the Australian Research Council's Discovery Projects Scheme (projects DP110101934 and DP140102655). We thank Alec Zwart and Xiaolu Lu.  ... 
doi:10.1145/2766462.2767728 dblp:conf/sigir/BaileyMST15 fatcat:ctaphtd565akbofhoxxkl36cwq

IR system evaluation using nugget-based test collections

Virgil Pavlu, Shahzad Rajput, Peter B. Golbus, Javed A. Aslam
2012 Proceedings of the fifth ACM international conference on Web search and data mining - WSDM '12  
The development of information retrieval systems such as search engines relies on good test collections, including assessments of retrieved content.  ...  We then show how these inferred relevance assessments can be used to perform IR system evaluation, and we discuss in particular reusability and scalability.  ...  test collection methodology to IR system evaluation.  ... 
doi:10.1145/2124295.2124343 dblp:conf/wsdm/PavluRGA12 fatcat:3tzpysivnjdk3aerpj6plnwbhe

Scaling IR-system evaluation using term relevance sets

Einat Amitay, David Carmel, Ronny Lempel, Aya Soffer
2004 Proceedings of the 27th annual international conference on Research and development in information retrieval - SIGIR '04  
This paper describes an evaluation method based on Term Relevance Sets (Trels) that measures an IR system's quality by examining the content of the retrieved results rather than by looking for pre-specified  ...  Moreover, this method can evaluate a system's effectiveness on an updatable "live" collection, or on collections derived from different data sources.  ...  INTRODUCTION The evaluation of information retrieval (IR) systems is the process of assessing how well a system meets the information needs of its users.  ... 
doi:10.1145/1008992.1008997 dblp:conf/sigir/AmitayCLS04 fatcat:creiqewfyfc57jitoxrnlmzmee

Evaluation of the Impact on the Environment at Building and Reconstruction of Motorways Using the System Analysis Method

Viktoriia Khrutba, Yevheniia Anpilova, Vitalina Lukianova, Iryna Kotsiuba, Lesia Kriukovska, Oksana Spasichenko
2021 Aplinkos tyrimai, inzinerija ir vadyba / Environmental Research, Engineering and Management  
From the point of view of the system approach, the interrelation in the system "highway repair – environment" was investigated, which allowed systematizing the main aspects of environmental impact during  ...  – Reshetylivka and its impact on the environment.  ...  The objective of the work is the development of a system model of the cause-effect relationship for evaluating the impact of work connected to reconstruction (maintenance) of the motorway on the environment  ... 
doi:10.5755/j01.erem.77.1.27887 fatcat:cljob7mi5bcqjolxg45ceh4qpm

An uncertainty-aware query selection model for evaluation of IR systems

Mehdi Hosseini, Ingemar J. Cox, Natasa Milic-Frayling, Milad Shokouhi, Emine Yilmaz
2012 Proceedings of the 35th international ACM SIGIR conference on Research and development in information retrieval - SIGIR '12  
We demonstrate the effectiveness of the algorithm on two TREC test collections as well as a test collection of an online search engine with 1000 queries.  ...  Our experimental results show that the queries chosen by Adaptive produce reliable performance ranking of systems.  ...  Generally, the query selection methods have been criticized for their lack of generalization to previously unseen systems and multiple evaluation metrics. Our Adaptive al-  ... 
doi:10.1145/2348283.2348403 dblp:conf/sigir/HosseiniCMSY12 fatcat:kjqf6xwedfgn7fvmvheacucn5i
« Previous Showing results 1 — 15 out of 652,005 results