Filters








731 Hits in 5.8 sec

Magnitudes of Relevance: Relevance Judgements, Magnitude Estimation, and Crowdsourcing

Falk Scholer, Eddy Maddalena, Stefano Mizzaro, Andrew Turpin
2014 NTCIR Conference on Evaluation of Information Access Technologies  
By performing a crowdsourcing experiment, we show that magnitude estimation relevance judgements are consistent with ordinal relevance ones; we study the difference of using a bounded or an unbounded scale  ...  We report on a preliminary investigation on using magnitude estimation for gathering document relevance judgements, as commonly used in test collectionbased evaluation of information retrieval systems.  ...  Are crowdsourced, document-level relevance judgements obtained using the magnitude estimation technique consistent with expert judgements obtained on an ordinal scale? 2.  ... 
dblp:conf/ntcir/ScholerMMT14 fatcat:3vtxiua5cja37n5ohhidvse524

Functional Representation of Prototypes in LVQ and Relevance Learning [chapter]

Friedrich Melchert, Udo Seiffert, Michael Biehl
2016 Advances in Intelligent Systems and Computing  
Held yearly, the objective of BNAIC is to promote and disseminate recent research developments in Artificial Intelligence within Belgium, Luxembourg and the Netherlands.  ...  We received 93 submissions, consisting of 24 regular papers, 47 short papers, 11 demonstration abstracts and 11 thesis abstracts.  ...  This work successfully builds on previous theories from the field of new media and linguistics.  ... 
doi:10.1007/978-3-319-28518-4_28 fatcat:uwxvq6txmrba3ajulmblafgh2a

Results of Evaluation Runs and Data Analysis in the LibRank project

Christiane Behnert, Kim Plassmeier
2016 Zenodo  
We present the results of the data analysis to every evaluation run and discuss possible reasons for the results in separate sections before we summarize our findings.  ...  Therefore we removed all tasks for which the difference of the maximal non-relevant judgement and the minimal relevant judgement exceeded a threshold of ten points.  ...  Thus, eliciting relevance judgments via magnitude estimation (Maddalena, Mizzaro, Scholer, & Turpin, 2015) or via preference judgments (Carterette, Bennett, & Dumais, 2008) might decrease the effect  ... 
doi:10.5281/zenodo.4911463 fatcat:gv7dtzayg5chthzxvmwmqc64hu

Assessing top-k preferences [article]

Charles L. A. Clarke, Alexandra Vtyurina, Mark D. Smucker
2021 arXiv   pre-print
To measure the performance of a ranker, we compare its output to this preferred ordering by applying a rank similarity measure.We demonstrate the practical feasibility of this approach by crowdsourcing  ...  In this paper, we explore the assessment process for partial preference judgments, with the aim of identifying and ordering the top items in the pool, rather than fully ordering the entire pool.  ...  [27] crowdsource relevance magnitudes through a process in which assessors view a series of documents and estimate relevance relative to the previously seen document. eir results call into question  ... 
arXiv:2007.11682v2 fatcat:lkdxeb5dgbe5zk6vo532bad5ge

Building a large-scale corpus for evaluating event detection on twitter

Andrew J. McMinn, Yashar Moshfeghi, Joemon M. Jose
2013 Proceedings of the 22nd ACM international conference on Conference on information & knowledge management - CIKM '13  
We then use crowdsourcing to gather relevance judgements, and discuss the quality of results, including how we ensured integrity and prevented spam.  ...  This is partially due to a number of issues associated with the creation of Twitter corpora, including restrictions on the distribution of the tweets and the difficultly of creating relevance judgements  ...  We then use crowdsourcing to evaluate if a candidate event fits our definition of event, and create relevance judgements for each of the events.  ... 
doi:10.1145/2505515.2505695 dblp:conf/cikm/McMinnMJ13 fatcat:biw36szlzncdjllbjerwffbsiq

Using The Wisdom Of The Crowd To Predict Popular Music Chart Success

Dennis M. Steininger, Simon Gatzemeier
2013 European Conference on Information Systems  
However, this predictive relevance is bound to certain conditions, namely the composition of the crowd, the underlying chart and market mechanisms and the novelty of the musical material.  ...  With respect to the on-going digital transformation and shift in power from organizations to consumers, we leverage the principles of crowdsourcing to build a prediction model for understanding chart success  ...  The results from the estimation have shown, that the path between the Overall Affective Response and Chart Success has predictive relevance ( ).  ... 
dblp:conf/ecis/SteiningerG13 fatcat:d6eneito6rcjbmyh255xdkr57m

Crowdsourcing Multimedia QoE Evaluation: A Trusted Framework

Chen-Chi Wu, Kuan-Ta Chen, Yu-Chun Chang, Chin-Laung Lei
2013 IEEE transactions on multimedia  
We demonstrate the effectiveness and efficiency of the proposed framework by a comparison with MOS.  ...  Crowdsourcing has emerged in recent years as a potential strategy to enlist the general public to solve a wide variety of tasks.  ...  In [29] , Horton and Chilton investigated the relationship of wages and task difficulty, and proposed a model to estimate participants' reservation wage in crowdsourcing tasks.  ... 
doi:10.1109/tmm.2013.2241043 fatcat:o6a5kekhnjarvi4b2gjn6zei3q

Learning to rank by aggregating expert preferences

Maksims N. Volkovs, Hugo Larochelle, Richard S. Zemel
2012 Proceedings of the 21st ACM international conference on Information and knowledge management - CIKM '12  
When applied to crowdsourcing and meta-search benchmarks, our new algorithm improves on state-of-the-art preference aggregation methods.  ...  We present a general treatment of the problem of aggregating preferences from several experts into a consensus ranking, in the context where information about a target ranking is available.  ...  The dataset is a collection of binary (relevant or not relevant) judgements from 762 workers for 19,033 documents.  ... 
doi:10.1145/2396761.2396868 dblp:conf/cikm/VolkovsLZ12 fatcat:y2sgypmmjnfwhhluq2qtm44vt4

Constructing Better Evaluation Metrics by Incorporating the Anchoring Effect into the User Model

Nuo Chen, Fan Zhang, Tetsuya Sakai
2022 Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval  
Previous studies showed that the anchoring effect can influence the relevance judgement of a document.  ...  In the framework, the magnitude of the anchoring effect is related to the quality of the previous document.  ...  [28] also observed the anchoring effect in relevance judgements of crowdsourced labels.  ... 
doi:10.1145/3477495.3531953 fatcat:e3ovot2hzfbb3afaouskmueosm

The Effects of Performance-Contingent Financial Incentives in Online Labor Markets

Ming Yin, Yiling Chen, Yu-An Sun
2013 PROCEEDINGS OF THE THIRTIETH AAAI CONFERENCE ON ARTIFICIAL INTELLIGENCE AND THE TWENTY-EIGHTH INNOVATIVE APPLICATIONS OF ARTIFICIAL INTELLIGENCE CONFERENCE  
We find that the magnitude of performance-contingent financial rewards alone affects neither quality nor effort.  ...  However, when workers working on two tasks of the same type in a sequence, the change in the magnitude of the reward over the two tasks affects both.  ...  Estimated coefficients and standard errors are reported.  ... 
doi:10.1609/aaai.v27i1.8461 fatcat:7ow4cyfqkbbz5dy3xrq666fuum

An Image is Worth More than a Thousand Favorites: Surfacing the Hidden Beauty of Flickr Pictures [article]

Rossano Schifanella, Miriam Redi, Luca Aiello
2015 arXiv   pre-print
By gathering a large crowdsourced ground truth of aesthetics scores for Flickr images, we show that our method retrieves photos whose median perceived beauty score is equal to the most popular ones, and  ...  Attention concentrates on a relatively small number of popular items and neglecting the vast majority of content produced by the crowd.  ...  Judith Redi for her precious help and discussions.  ... 
arXiv:1505.03358v2 fatcat:fu4sq22my5febhkgbylmmyxki4

Everything you always wanted to know about a dataset: studies in data summarisation

Laura Koesten, Elena Simperl, Tom Blount, Emilia Kacprzak, Jeni Tennison
2019 International Journal of Human-Computer Studies  
In each study we carried out a qualitative analysis to identify key themes and commonly mentioned dataset attributes, which people consider when searching and making sense of data.  ...  We present two complementary studies: a data-search diary study with 69 students, which offers insight into the information needs of people searching for data; and a summarisation study, with a lab and  ...  Acknowledgements This project was supported by the European Union Horizon 2020 research and innovation programme under the Marie Skłodowska-Curie grant agreement No 642795,and No 780247 and by the EPSRC  ... 
doi:10.1016/j.ijhcs.2019.10.004 fatcat:glnaoezefrgi3cbpn5pcdzqusu

Estimating Return on Impact of Misinformation Intervention [article]

Sahil Shah, R.J. Cordes, Pat Scannell, Alex Ruiz, Scott David
2022 Zenodo  
The incentivization of response reporting and information sharing is discussed, and a series of recommendations for continued work are provided.  ...  This white paper offers an operational approach toward defining the financial impacts of misinformation and explores potential approaches to analysing and integrating accounting of response costs, opportunity  ...  For a deeper discussion of this topic, see Judgement under Uncertainty: Heuristics and Biases, by Amos Tversky and Daniel Kahneman, and Psychology of Intelligence Analysis, by RJ Heuer.  ... 
doi:10.5281/zenodo.6529737 fatcat:aqavfxqvl5gjbj6ggincerr354

Predictive modeling for trustworthiness and other subjective text properties in online nutrition and health communication

Janne Kauttonen, Jenni Hannukainen, Pia Tikka, Jyrki Suomala, Amira M. Idrees
2020 PLoS ONE  
With a novel combination of crowdsourcing, behavioral analysis, natural language processing and predictive modeling, our study contributes to the automated identification of reliable and high-quality online  ...  We then estimated individual biases and consensus ratings that were used in training regression models.  ...  With the attribute of subjective it is acknowledged that these estimates ultimately depend on personal judgement, knowledge, personal values and interests of raters.  ... 
doi:10.1371/journal.pone.0237144 pmid:32760095 fatcat:b7ie4jqmfzhwrev6whv6awry7m

News vertical search using user-generated content

Richard McCreadie
2012 SIGIR Forum  
Many of these judgements are then combined together to form a test collection. This consists of the relevance judgements, the corpus of documents and the queries used.  ...  Hence, the crowdsourced assessment task is to judge each of the pooled blog posts as relevant, possibly relevant or not relevant to a newswire article (facilitating relevance evaluation), and also to suggest  ...  Both Votes and RWA use a textual representation of a news story a to retrieve recent and related blog posts or blog feeds for a news story.  ... 
doi:10.1145/2492189.2492202 fatcat:wuha3gotmnffnbqhrdltooys5m
« Previous Showing results 1 — 15 out of 731 results