Filters








14 Hits in 9.0 sec

Crowdsourcing for search evaluation

Vitor R. Carvalho, Matthew Lease, Emine Yilmaz
2011 SIGIR Forum  
However, no dataset currently exists upon which approaches to news query classification can be evaluated and compared.  ...  Ground truths based on partially ordered lists were developed to cope with problems regarding relevance judgment, but they require such man-power to generate that the official MIREX evaluations had to  ...  Labs and in part by the University of Delaware Research Foundation.  ... 
doi:10.1145/1924475.1924481 fatcat:56ywunsa6vgdvmjmoulohmq5ye

The challenging task of summary evaluation: an overview

Elena Lloret, Laura Plaza, Ahmet Aker
2017 Language Resources and Evaluation  
To perform an adequate evaluation is of great relevance to ensure that automatic summaries can be useful for the context and/or application they are generated for.  ...  In this article, a critical and historical analysis of evaluation metrics, methods, and datasets for automatic summarization systems is presented, where the strengths and weaknesses of evaluation efforts  ...  ), and the Universidad Nacional de Educación a Distancia through the project "Modelado y síntesis automática de opiniones de usuario en redes sociales" (2014-001-UNED-PROY).  ... 
doi:10.1007/s10579-017-9399-2 fatcat:tduxzlv2hfbfvd6evzoxn5xibu

News vertical search using user-generated content

Richard McCreadie
2012 SIGIR Forum  
Classical IR evaluation has centred around human relevance judgements. This is where a human looks at a document given a query and marks it relevant or not to that query.  ...  Crowdsourcing provides a potentially fast and cheap means to generate assessments for our datasets (Alonso et al., 2008) .  ...  windows r and w.  ... 
doi:10.1145/2492189.2492202 fatcat:wuha3gotmnffnbqhrdltooys5m

On the impact of domain expertise on query formulation, relevance assessment and retrieval performance in clinical settings

Lynda Tamine, Cecile Chouquet
2017 Information Processing & Management  
This article focuses on the extent to which expertise can impact clinical query formulation, document relevance assessment and retrieval performance in the context of tailoring retrieval models and systems  ...  The findings of this study presents opportunities for the design of personalized health-related IR systems, but also for providing insights about the evaluation of such systems.  ...  User studies using crowdsourcing platforms Crowdsourcing has become a powerful tool for obtaining labels for IR system development and evaluation (Lease and Yelmaz, 2011) .  ... 
doi:10.1016/j.ipm.2016.11.004 fatcat:ekgnxyvzurcbha35n64jxs7o5a

Structuring the world's knowledge: Socio-technical processes and data quality in Wikidata

Alessandro Piscopo
2019 Figshare  
In particular, it makes a threefold contribution: (i.) it evaluates two previously uncovered aspects of the quality of Wikidata, i.e. provenance and its ontology; (ii.) it is the first to investigate the  ...  Finally, two roles emerge from the editing patterns of Wikidata users, leaders and contributors. Leaders perform more edits and have a more prominent role within the community.  ...  Microtasks Total judgements Relevance and authoritativeness evaluation The next sections report the findings of the reference evaluation.  ... 
doi:10.6084/m9.figshare.10998791 fatcat:n7qfv5yefndrvfko24uy5rwtwm

Structuring the world's knowledge: Socio-technical processes and data quality in Wikidata

Alessandro Piscopo
2019 Figshare  
In particular, it makes a threefold contribution: (i.) it evaluates two previously uncovered aspects of the quality of Wikidata, i.e. provenance and its ontology; (ii.) it is the first to investigate the  ...  Finally, two roles emerge from the editing patterns of Wikidata users, leaders and contributors. Leaders perform more edits and have a more prominent role within the community.  ...  Microtasks Total judgements Relevance and authoritativeness evaluation The next sections report the findings of the reference evaluation.  ... 
doi:10.6084/m9.figshare.10998791.v1 fatcat:7apohgf4zvbnfa426zgsekq6sy

Explicit web search result diversification

Rodrygo L.T. Santos
2012 SIGIR Forum  
et al., , 2009a , and by conducting additional relevance assessments, e.g., through crowdsourcing (Alonso et al., 2008) .  ...  have no apparent bearing on topical relevance, such as query categories.  ... 
doi:10.1145/2492189.2492205 fatcat:g3f4j6r6ivhtzbm6mfi2zigsm4

Pretrained Transformers for Text Ranking: BERT and Beyond

Andrew Yates, Rodrigo Nogueira, Jimmy Lin
2021 Proceedings of the 14th ACM International Conference on Web Search and Data Mining  
The combination of transformers and self-supervised pretraining has, without exaggeration, revolutionized the fields of natural language processing (NLP), information retrieval (IR), and beyond.  ...  In the context of text ranking, these models produce high quality results across many domains, tasks, and settings.  ...  Although transformer architectures and pretraining techniques are recent innovations, many aspects of how they are applied to text ranking are relatively well understood and represent mature techniques  ... 
doi:10.1145/3437963.3441667 fatcat:6teqmlndtrgfvk5mneq5l7ecvq

A Multi-Lingually Applicable Journalist Toolset For The Big-Data Era

G. Kiomourtzis, G. Giannakopoulos, V. Karkaletsis, A. Kosmopoulos
2016 Zenodo  
No 645886.  ...  Agreements No.  ...  Preprocessing the corpus might produce topics that are more concise and relevant.  ... 
doi:10.5281/zenodo.1242850 fatcat:nfkqg7jhjffdvgezdjzc6xxppa

Ranking for Web Data Search Using On-The-Fly Data Integration

Daniel Markus Herzig
2014
These characteristics hamper the adoption of structured Web data for search and require new methods for ranking where ambiguity and vagueness challenge the assessment of relevance.  ...  We propose solutions regarding these research questions and experimentally analyze and evaluate them against the latest baselines. The results show advances beyond the state-of-the-art. i  ...  How we use hybrid queries and data in our ranking approach is explained in Section 4. 4 . Related work is discussed in Section 4.5 and related and prior approaches are used in our evaluation.  ... 
doi:10.5445/ksp/1000037230 fatcat:xqkkq7otm5e7tltog2dcoc64xe

Message from the general chair

Benjamin C. Lee
2015 2015 IEEE International Symposium on Performance Analysis of Systems and Software (ISPASS)  
Learning-based Multi-Sieve Co-reference Resolution with Knowledge Lev Ratinov and Dan Roth Saturday 11:00am-11:30am -202 A (ICC) We explore the interplay of knowledge and structure in co-reference resolution  ...  Joint Learning for Coreference Resolution with Markov Logic Resolving "This-issue" Anaphora Varada Kolhatkar and Graeme Hirst Saturday 12:00pm-12:30pm -202 A (ICC) We annotate and resolve a particular  ...  We evaluate the said metrics through a user-assessed quality of the generated two-liners.  ... 
doi:10.1109/ispass.2015.7095776 dblp:conf/ispass/Lee15 fatcat:ehbed6nl6barfgs6pzwcvwxria

A Survey of App Store Analysis for Software Engineering

William Martin, Federica Sarro, Yue Jia, Yuanyuan Zhang, Mark Harman
2017 IEEE Transactions on Software Engineering  
, software design, security and testing.  ...  This survey describes and compares the areas of research that have been explored thus far, drawing out common aspects, trends and directions future research should take to address open problems and challenges  ...  This research was supported by EPRSC (DAASE grant no. EP/J017515).  ... 
doi:10.1109/tse.2016.2630689 fatcat:tuqtkqnzordklgq2thihhc6sxy

Proceedings for the 3rd Shaw-IAU Workshop on Astronomy for Education: What Everybody Should Know about Astronomy Education [article]

Asmita Bhandare, Giuliana Giobbi, Colm Larkin, Rebecca Sanderson, Eduardo Penteado, Niall Deacon, Gwen Sanderson, Anna Sippel
2021 Zenodo  
, with additional support from the National Aeronautics and Space Administration under Award No. NNX AC A.  ...  and Hia-Ced O'odham Nations in Arizona; and the Massachusett, Nipmuk, and Wampanoag Nations in Massachusetts.  ...  On the other hand, the IR m-os appear in all grades and include more topics and higher cognitive processes.  ... 
doi:10.5281/zenodo.5768700 fatcat:m7ltmzzmjbcr3inrdnweemstly

Dr. Eric Archambault Profile

2016 Against the Grain  
IRs and Creative Commons.  ...  (IR).  ...  For as Pollan points out, when evaluating cost, we need to incorporate full costs in our assessments.  ... 
doi:10.7771/2380-176x.7476 fatcat:c6dlj5luf5auxgy3ajzowrk4pq