139,108 Hits in 6.6 sec

Exploring Cost-Effective Approaches to Human Evaluation of Search Engine Relevance [chapter]

Kamal Ali, Chi-Chao Chang, Yunfang Juan
2005 Lecture Notes in Computer Science  
measure effectiveness relative to other search engines.  ...  In this paper, we examine novel and less expensive methods for search engine evaluation that do not rely on document relevance judgments.  ...  Acknowledgments: Thank you to Jan Pedersen at Yahoo! and the editorial team for hundreds of hours of work on judging search results.  ... 
doi:10.1007/978-3-540-31865-1_26 fatcat:ndvl2jjehvabpj2lqfte7xfv7y

Explaining User Performance in Information Retrieval: Challenges to IR Evaluation [chapter]

Kalervo Järvelin
2009 Lecture Notes in Computer Science  
The paper makes three points of significance for IR research: (1) The Cranfield paradigm of IR evaluation seems to lose power when one looks at human instead of system performance. (2) Searchers using  ...  Evaluation of output alone is insufficient to explain searcher behavior.  ...  Acknowledgement This research was funded by the Academy of Finland grants #120996 and #124131.  ... 
doi:10.1007/978-3-642-04417-5_28 fatcat:d4ttunpujrblbairoqlohvuxna

Using the Crowd to Improve Search Result Ranking and the Search Experience

Yubin Kim, Kevyn Collins-Thompson, Jaime Teevan
2016 ACM Transactions on Intelligent Systems and Technology  
This paper explores how crowdsourcing can be used at query time to augment key stages of the search pipeline. We first explore the use of crowdsourcing to improve search result ranking.  ...  While search tools that rely on human intervention will never be able to respond as quickly as current search engines do, recent research suggests that there are scenarios where a search engine could take  ...  To make richer use of the crowd, we also explore several crowd-based approaches that change the existing search experience.  ... 
doi:10.1145/2897368 fatcat:symwsreokzcsvg6oko6lzi7h7a

Automated evaluation of search engine performance via implicit user feedback

Himanshu Sharma, Bernard J. Jansen
2005 Proceedings of the 28th annual international ACM SIGIR conference on Research and development in information retrieval - SIGIR '05  
Measuring the information retrieval effectiveness of Web search engines can be expensive if human relevance judgments are required to evaluate search results.  ...  Web search engines can use human evaluation of search results without the expense of human evaluators.  ...  engine performance by reducing the cost and time of evaluating retrieved results.  ... 
doi:10.1145/1076034.1076172 dblp:conf/sigir/SharmaJ05 fatcat:xnp3irh7vra5pfbysmgxvpkotu

Mining Implicit Relevance Feedback from User Behavior for Web Question Answering [article]

Linjun Shou, Shining Bo, Feixiang Cheng, Ming Gong, Jian Pei, Daxin Jiang
2020 arXiv   pre-print
In practice, this work has proved effective to substantially reduce the human labeling cost for the QA service in a global commercial search engine, especially for languages with low resources.  ...  One principled idea is to mine implicit relevance feedback from user behavior recorded in search engine logs.  ...  Our extensive experiments demonstrate the effectiveness of this approach in improving the performance of QA models and thus reducing the human labeling cost.  ... 
arXiv:2006.07581v1 fatcat:x24sblby6jhjxhrywdqmkdyt4q

Cost and benefit analysis of mediated enterprise search

Mingfang Wu, James A. Thom, Andrew Turpin, Ross Wilkinson
2009 Proceedings of the 2009 joint international conference on Digital libraries - JCDL '09  
evaluation also indicates the type of queries to be selected from a query log for evaluating an enterprise search engine.  ...  We propose a method that evaluates the cost and benefit for a service provider of using a mediated search engine -in particular, where domain experts intervene on the ranking of the search results from  ...  COST AND BENEFIT ANALYSIS In this section, we develop a cost and benefit analysis method to evaluate the cost effectiveness of the mediated enterprise search approach.  ... 
doi:10.1145/1555400.1555445 dblp:conf/jcdl/WuTTW09 fatcat:foofiza5c5g3nguovqoukrkpjy

Pre-trained Language Model based Ranking in Baidu Search [article]

Lixin Zou, Shengqiang Zhang, Hengyi Cai, Dehong Ma, Suqi Cheng, Daiting Shi, Zhifan Zhu, Weiyue Su, Shuaiqiang Wang, Zhicong Cheng, Dawei Yin
2021 arXiv   pre-print
search engine system.  ...  As the heart of a search engine, the ranking system plays a crucial role in satisfying users' information demands.  ...  the efficiency and effectiveness of PLM-based ranking schema in the real-world search engine system. • Relevance-oriented Pre-training.  ... 
arXiv:2105.11108v3 fatcat:dbvj65ugovaani4hsiwtl6bcdi

Introducing natural language program analysis

Lori Pollock, K. Vijay-Shanker, David Shepherd, Emily Hill, Zachary P. Fry, Kishen Maloor
2007 Proceedings of the 7th ACM SIGPLAN-SIGSOFT workshop on Program analysis for software tools and engineering - PASTE '07  
We then present an overview of our extraction process, program representation, and a set of tools we have developed using this natural language program analysis.  ...  This research group presentation focuses on our work in extracting and utilizing natural language clues from source code to improve software maintenance tools.  ...  Building effective software tools is important to reduce these high maintenance costs.  ... 
doi:10.1145/1251535.1251538 dblp:conf/paste/PollockVSHFM07 fatcat:gqerfkoixjfldgtkx2a72f4ss4

Crowdsourcing for search evaluation and social-algorithmic search

Matthew Lease, Omar Alonso
2012 Proceedings of the 35th international ACM SIGIR conference on Research and development in information retrieval - SIGIR '12  
Today, Internet-based access to 24/7 online human crowds has led to a renaissance of research in human computation and the advent of crowdsourcing.  ...  This tutorial introduces opportunities and challenges of human computation and crowdsourcing, particularly for search evaluation and developing hybrid search solutions that integrate human computation  ...  Many Web search engines reportedly use large editorial staffs to judge the relevance of web pages for queries in an evaluation set, but his is expensive and has limited scalability.  ... 
doi:10.1145/2348283.2348530 dblp:conf/sigir/LeaseA12 fatcat:mjcbgbnlxrch7nb2xu2vgng57q

CapturingCollabportunities: A method to evaluate collaboration opportunities in information search using pseudocollaboration

Roberto González-Ibáñez, Chirag Shah, Ryen W. White
2014 Journal of the Association for Information Science and Technology  
We developed a technique called pseudocollaboration to evaluate the benefits and costs of collabportunities through simulations.  ...  a largescale search engine with search sessions of thousands of searchers.  ...  Acknowledgments The work described in this article was in part supported by the Institute of Museum and Library Services (IMLS) Early Career Development grant # RE-04-12-0105-12.  ... 
doi:10.1002/asi.23288 fatcat:6isney2ag5dlnpad2kenk3zgwq

Architecture-driven quality requirements prioritization

Anne Koziolek
2012 2012 First IEEE International Workshop on the Twin Peaks of Requirements and Architecture (TwinPeaks)  
Our method uses automated design space exploration based on quantitative quality evaluation of software architecture models.  ...  To decide for meaningful, feasible quality requirements and trade them off with functional requirements, tighter integration of software architecture evaluation and requirements prioritization is necessary  ...  In the field of requirements engineering, numerous approaches to handle quality requirements have been suggested [21] , [22] .  ... 
doi:10.1109/twinpeaks.2012.6344554 dblp:conf/re/Koziolek12 fatcat:4d5j34f4gbfvferq4ryf7p63wm

Counterfactual Estimation and Optimization of Click Metrics for Search Engines [article]

Lihong Li and Shunbao Chen and Jim Kleban and Ankur Gupta
2014 arXiv   pre-print
The key challenge is the counterfactual nature: in the case of Web search, any change to a component of the search engine may result in a different search result page for the same query, but we normally  ...  This approach effectively allows one to run (potentially infinitely) many A/B tests offline from search log, making it possible to estimate and optimize online metrics quickly and inexpensively.  ...  INTRODUCTION The standard approach to evaluating ranking quality of a search engine is to evaluate its ranking results on a set of human-labeled examples and compute relevance metrics like mean average  ... 
arXiv:1403.1891v2 fatcat:fb6k5zsulbb23jl3tnd5is4pn4

Explore click models for search ranking

Dong Wang, Weizhu Chen, Gang Wang, Yuchen Zhang, Botao Hu
2010 Proceedings of the 19th ACM international conference on Information and knowledge management - CIKM '10  
Yet, few works have been conducted to explore the use of click model to help web search ranking.  ...  Recent advances in click model have positioned it as an effective approach to estimate document relevance based on user behavior in web search.  ...  These click models have been considered as one of the most effective approaches to interpret user clicks and infer search relevance, and recent advances in click models have moved forward aggressively.  ... 
doi:10.1145/1871437.1871636 dblp:conf/cikm/WangCWZH10 fatcat:iuhl3fptqjbm5dydb5a5mdnynm

Efficient Test Collection Construction via Active Learning [article]

Md Mustafizur Rahman, Mucahid Kutlu, Tamer Elsayed, Matthew Lease
2018 arXiv   pre-print
To create a new IR test collection at minimal cost, we must carefully select which documents merit human relevance judgments.  ...  Results demonstrate the effectiveness of our approach, coupled with further analysis showing how varying relevance scarcity, within and across collections, impacts findings.  ...  Even commercial search engines, despite their query logs, reportedly continue to depend on large teams of human Permission to make digital or hard copies of all or part of this work for personal or classroom  ... 
arXiv:1801.05605v2 fatcat:ssaz5gvat5h43njyf5difo7vju

Promoting transactions for A/E/C product information

Ken-Yu Lin, Lucio Soibelman
2006 Automation in Construction  
an existing search engine their capabilities in gathering additional product information.  ...  Therefore, this researcher has developed a knowledge-supported approach to tackle the recognized issues to obtain an increased amount of product information from the available market, as a way to promote  ...  The significance of efforts and resources needed to search for relevant product information is associated with the type of product being evaluated and the frequency with which a product is acquired.  ... 
doi:10.1016/j.autcon.2005.09.008 fatcat:hd3eod7ocvbfbh7cq2mygw2o44
« Previous Showing results 1 — 15 out of 139,108 results