Filters








85,099 Hits in 6.5 sec

IR system evaluation using nugget-based test collections

Virgil Pavlu, Shahzad Rajput, Peter B. Golbus, Javed A. Aslam
2012 Proceedings of the fifth ACM international conference on Web search and data mining - WSDM '12  
We propose a new method for relevance assessment based on relevant information, not relevant documents.  ...  The development of information retrieval systems such as search engines relies on good test collections, including assessments of retrieved content.  ...  To this end, we constructed two separate test collections based on wellstudied collections produced by previous TREC tracks.  ... 
doi:10.1145/2124295.2124343 dblp:conf/wsdm/PavluRGA12 fatcat:3tzpysivnjdk3aerpj6plnwbhe

A nugget-based test collection construction paradigm

Shahzad Rajput, Virgil Pavlu, Peter B. Golbus, Javed A. Aslam
2011 Proceedings of the 20th ACM international conference on Information and knowledge management - CIKM '11  
documents found by TREC assessors-as well as up to four times more additional relevant documents.  ...  Starting with a few relevant "nuggets" of information manually extracted from existing TREC corpora, we implement and test a methodology that finds and correctly assesses the vast majority of relevant  ...  Acknowledgment: This material is based upon work supported by the National Science Foundation under Grant No. IIS-1017903.  ... 
doi:10.1145/2063576.2063861 dblp:conf/cikm/RajputPGA11 fatcat:cj7jsyqi7jfyfom6wsxnvggqb4

Pseudo test collections for learning web search ranking functions

Nima Asadi, Donald Metzler, Tamer Elsayed, Jimmy Lin
2011 Proceedings of the 34th international ACM SIGIR conference on Research and development in Information - SIGIR '11  
Test collections are the primary drivers of progress in information retrieval.  ...  However, manual construction of test collections tends to be slow, labor-intensive, and expensive.  ...  RELATED WORK There are two steps involved in constructing pseudo test collections-sampling pseudo queries and inferring pseudo relevance judgments for the queries.  ... 
doi:10.1145/2009916.2010058 dblp:conf/sigir/AsadiMEL11 fatcat:e6pkrqxunzffliik7tu3dgiff4

Classifying Document Titles Based on Information Inference [chapter]

Dawei Song, Peter Bruza, Zi Huang, Raymond Y. K. Lau
2003 Lecture Notes in Computer Science  
Information inference can be performed on the HAL spaces via computing information flow between vectors or combination vectors.  ...  We propose an intelligent document title classification agent based on a theory of information inference.  ...  Acknowledgements The work reported in this paper has been funded in part by the Cooperative Research Centres Program through the Department of the Prime Minister and Cabinet of Australia.  ... 
doi:10.1007/978-3-540-39592-8_41 fatcat:5kjhwlkt6verpjlr7kcqz3gelu

Computer-Assisted Relevance Assessment: A Case Study of Updating Systematic Medical Reviews

Noha S. Tawfik, Marco Spruit
2020 Applied Sciences  
These efforts can be significantly reduced by applying computer-assisted techniques to identify relevant studies.  ...  The primary outcome of interest was to compare the performance levels achieved when judging full abstracts versus single sentences accompanied by Natural Language Inference labels.  ...  test collections for information retrieval tasks.  ... 
doi:10.3390/app10082845 fatcat:er2jeueh6rei7gsfdslttfeeki

QBSUM: a Large-Scale Query-Based Document Summarization Dataset from Real-world Applications [article]

Mingjun Zhao, Shengli Yan, Bang Liu, Xinwang Zhong, Qian Hao, Haolan Chen, Di Niu, Bowei Long, Weidong Guo
2020 arXiv   pre-print
Query-based document summarization aims to extract or generate a summary of a document which directly answers or is relevant to the search query.  ...  We also propose multiple unsupervised and supervised solutions to the task and demonstrate their high-speed inference and superior performance via both offline experiments and online A/B tests.  ...  The summary Y is constructed by concatenating the selected text pieces in the order they present in the document.  ... 
arXiv:2010.14108v1 fatcat:haz6ygywuzddblive4v2igw2di

Inferring query models by computing information flow

P. D. Bruza, D. Song
2002 Proceedings of the eleventh international conference on Information and knowledge management - CIKM '02  
Information flow is a reflection of how strongly w is informationally contained within the query Q. In other words, the basis of the query model generation is information inference.  ...  Experimental results are provided showing the HAL-based information flow model be superior to query models computed via Markov chains, and seems to be as effective as a probabilistically motivated relevance  ...  ACKNOWLEDGEMENTS The work reported in this paper has been funded in part by the Cooperative Research Centres Program through the Department of the Prime Minister and Cabinet of Australia.  ... 
doi:10.1145/584792.584837 dblp:conf/cikm/BruzaS02 fatcat:ftxygfhmejch5naloijw6ov4vu

Efficient Test Collection Construction via Active Learning [article]

Md Mustafizur Rahman, Mucahid Kutlu, Tamer Elsayed, Matthew Lease
2018 arXiv   pre-print
To create a new IR test collection at minimal cost, we must carefully select which documents merit human relevance judgments.  ...  Shared task campaigns such as NIST TREC determine this by pooling search results from many participating systems (and often interactive runs as well), thereby identifying the most likely relevant documents  ...  Assume we have a document collection X of m documents (represented by extracted features). Let y i j denote the binary relevance judgment for <document i, topic j>.  ... 
arXiv:1801.05605v2 fatcat:ssaz5gvat5h43njyf5difo7vju

Relevance and Effort

Emine Yilmaz, Manisha Verma, Nick Craswell, Filip Radlinski, Peter Bailey
2014 Proceedings of the 23rd ACM International Conference on Conference on Information and Knowledge Management - CIKM '14  
Relevance judgments sit at the core of test collection construction, and are assumed to model the utility of documents to real users.  ...  Information retrieval relevance judges are trained to search for evidence of relevance when assessing documents.  ...  [34] studied disagreements between judgments in test collections by identifying judged duplicate documents.  ... 
doi:10.1145/2661829.2661953 dblp:conf/cikm/YilmazVCRB14 fatcat:z3gqanz7v5gy7pr4y774es3jdm

Personalized Web Search via Query Expansion based on User's Local Hierarchically-Organized Files

Gianluca Moro, Roberto Pasolini, Claudio Sartori
2017 Proceedings of the 9th International Joint Conference on Knowledge Discovery, Knowledge Engineering and Knowledge Management  
A bag of keywords is extracted for each directory from text documents within it.  ...  We can infer the topic of each query and expand it by adding the corresponding keywords, in order to obtain a more targeted formulation.  ...  From this collection of documents already subdivided by topics, we can extract representative keywords for each topic, infer the topic of each query issued by the user and expand it by adding keywords  ... 
doi:10.5220/0006486401570164 dblp:conf/ic3k/MoroP017 fatcat:hcls7q6y5nb7dpkjgz2yy4ettq

Inferring query models by computing information flow

P. D. Bruza, D. Song
2002 Proceedings of the eleventh international conference on Information and knowledge management - CIKM '02  
Inferring query models by computing information flow.  ...  Inferring query models by computing information flow. Available from OpenAIR@RGU. [online]. Available from: http://openair.rgu.ac.uk Citation for the publisher's version: BRUZA, P.  ...  ACKNOWLEDGEMENTS The work reported in this paper has been funded in part by the Cooperative Research Centres Program through the Department of the Prime Minister and Cabinet of Australia.  ... 
doi:10.1145/584834.584837 fatcat:tp2esiyqyjc35js6zh7ileokbe

The University of Évora approach to QA@CLEF-2004

Paulo Quaresma, Luis Quintano, Irene Rodrigues, José Saias, Pedro D. Salgueiro
2004 Conference and Labs of the Evaluation Forum  
The system is based in two steps: for each question, a first information retrieval task selects a set of potentially relevant documents; then, each of these documents is analysed trying to obtain their  ...  The approach followed by the University of Évora team in order to build a system able to participate in the QA-CLEF task is described.  ...  So, we use an information retrieval system to obtain a set of relevant documents to make inferences only over the knowledge base created with the information conveyed by these documents.  ... 
dblp:conf/clef/QuaresmaQRSS04 fatcat:w6u6a6j6dbdknbe3n2tekhsv7e

Trialstreamer: Mapping and Browsing Medical Evidence in Real-Time [article]

Benjamin E. Nye, Ani Nenkova, Iain J. Marshall, Byron C. Wallace
2020 arXiv   pre-print
The system then attempts to infer which interventions were reported to work best by determining their relationship with identified trial outcome measures.  ...  Here we mainly describe the evidence extraction component; this extracts from biomedical abstracts key pieces of information that clinicians need when appraising the literature, and also the relations  ...  Acknowledgements This work was funded in part by the National Institutes of Health (NIH) under the National Library of Medicine (NLM) grant 2R01LM012086, and by the National Science Foundation (NSF) CA-REER  ... 
arXiv:2005.10865v1 fatcat:xnlfp6wg6re6te5sauco3zygui

Trialstreamer: Mapping and Browsing Medical Evidence in Real-Time

Benjamin Nye, Ani Nenkova, Iain Marshall, Byron C. Wallace
2020 Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations  
The system then attempts to infer which interventions were reported to work best by determining their relationship with identified trial outcome measures.  ...  Here we mainly describe the evidence extraction component; this extracts from biomedical abstracts key pieces of information that clinicians need when appraising the literature, and also the relations  ...  Acknowledgements This work was funded in part by the National Institutes of Health (NIH) under the National Library of Medicine (NLM) grant 2R01LM012086, and by the National Science Foundation (NSF) CA-REER  ... 
doi:10.18653/v1/2020.acl-demos.9 pmid:34136886 pmcid:PMC8204713 fatcat:72cl2fxr6vbnjk7hdj4zzhnctu

Overview of NTCIR-10

Hideo Joho, Tetsuya Sakai
2013 NTCIR Conference on Evaluation of Information Access Technologies  
This is an overview of NTCIR-10, the tenth sesquiannual workshop for the evaluation of Information Access technologies.  ...  This paper presents a brief history of NTCIR and overall statistics of NTCIR-10, followed by an introduction of eight evaluation tasks.  ...  The authors would like to thank the task organisers of all NTCIR-10 tasks for tremendous amount of effort devoted to run successful tasks, the task participants for their valuable contributions to the Information  ... 
dblp:conf/ntcir/JohoS13 fatcat:2xmg74l7pfg5jmwss7uvzlxvxu
« Previous Showing results 1 — 15 out of 85,099 results