1,462 Hits in 4.8 sec

Multi-Object Classification via Crowdsourcing With a Reject Option

Qunwei Li, Aditya Vempaty, Lav R. Varshney, Pramod K. Varshney
2017 IEEE Transactions on Signal Processing  
We consider the novel scenario where workers have a reject option so they may skip microtasks when they are unable or choose not to respond.  ...  Consider designing an effective crowdsourcing system for an M-ary classification task. Crowd workers complete simple binary microtasks whose results are aggregated to give the final result.  ...  binary microtask design with a reject option.  ... 
doi:10.1109/tsp.2016.2630038 fatcat:cc23wkuwkzh5zg2qb5wxjwcdsi

BPMN Task Instance Streaming for Efficient Micro-task Crowdsourcing Processes [chapter]

Stefano Tranquillini, Florian Daniel, Pavel Kucherbaev, Fabio Casati
2015 Lecture Notes in Computer Science  
The Business Process Model and Notation (BPMN) is a standard for modeling and executing business processes with human or machine tasks.  ...  We implement the necessary design and runtime support on top of CrowdFlower, demonstrate the viability of the approach via a case study, and report on a set of runtime performance experiments.  ...  rejection of results.  ... 
doi:10.1007/978-3-319-23063-4_23 fatcat:lhpmpfbsmfap5gtiqlntrqia4m

Prospect Theory Based Crowdsourcing for Classification in the Presence of Spammers [article]

Baocheng Geng, Qunwei Li, Pramod K. Varshney
2020 arXiv   pre-print
We consider the M-ary classification problem via crowdsourcing, where crowd workers respond to simple binary questions and the answers are aggregated via decision fusion.  ...  The workers have a reject option to skip answering a question when they do not have the expertise, or when the confidence of answering that question correctly is low.  ...  In Section II, we introduce the problem of multi-object classification via crowdsourcing with a reject option.  ... 
arXiv:1909.01463v2 fatcat:42tqotn7lbctrpfw7shsr5zlhq

A Discriminative Approach to Predicting Assessor Accuracy [chapter]

Hyun Joon Jung, Matthew Lease
2015 Lecture Notes in Computer Science  
Whereas prior assessor models have typically adopted a single generative approach, we formulate a discriminative, flexible feature-based model.  ...  Experiments using crowd assessor data from the NIST TREC 2011 Crowdsourcing Track show our model improves prediction accuracy by 26-36% across assessors, enabling 29-47% improved quality of relevance judgments  ...  In this study, we round off a probabilistic predictive value with a decision reject option as follows.  ... 
doi:10.1007/978-3-319-16354-3_17 fatcat:pesgwzsxfzasbfhtneg2p6eutm

Deep Partial Rank Aggregation for Personalized Attributes

Qianqian Xu, Zhiyong Yang, Zuyao Chen, Yangbangyan Jiang, Xiaochun Cao, Yuan Yao, Qingming Huang
2021 AAAI Conference on Artificial Intelligence  
First of all, we represent the pairwise annotations as a multi-graph.  ...  ., Shoes A is more comfortable than B) from different annotators on the crowdsourcing platforms, which is an emerging topic gaining increasing attention in recent years.  ...  Secondly, under the context of classification, the goal for including a reject option is to improve the robustness of the classifier and the reject option itself does not have a clear semantic meaning;  ... 
dblp:conf/aaai/Xu0CJC0H21 fatcat:cvizoj2xdbhn3fsxv4jar5gufq

EMTerms 1.0: A Terminological Resource for Crisis Tweets

Irina P. Temnikova, Carlos Castillo, Sarah Vieweg
2015 International Conference on Information Systems for Crisis Response and Management  
The terms are classified into 23 information-specific categories, by using a combination of expert annotations and crowdsourcing.  ...  The terms have been collected from a seed set of terms manually annotated by a linguist and an emergency manager from tweets broadcast during 4 crisis events.  ...  Crowdsource annotators rejected about 9.1% of the terms with confidence greater than 80%.  ... 
dblp:conf/iscram/TemnikovaCV15 fatcat:edypujxgirh5zf7w4hjzoa7duq

An Iterative Labeling Method for Annotating Fisheries Imagery [article]

Zhiyong Zhang, Pushyami Kaveti, Hanumant Singh, Abigail Powell, Erica Fruh, M. Elizabeth Clarke
2022 arXiv   pre-print
exploit crowdsourcing interfaces.  ...  Our results indicate that training with a small subset and iterating on that to build a larger set of labeled data allows us to converge to a fully annotated dataset with a small number of iterations.  ...  There are multiple options for labeling image data sets. The most simple utilize crowdsourcing on a marketplace such as Mechanical Turk [25] .  ... 
arXiv:2204.12934v2 fatcat:4hpttp3wwnaahe7afkb5soi3dq

Outsourcing labour to the cloud

Jonathan R. Corney, Carmen Torres Sanchez, A. Prasanna Jagadeesan, William C. Regli
2009 International Journal of Innovation and Sustainable Development  
Based on this, a generic classification for crowdsourcing tasks and a number of performance metrics are proposed.  ...  After discussing strengths and limitations, the paper concludes with an agenda for academic research in this new area.  ...  Flawed responses could be minimised by rewording of the task instructions or by reducing the number of options in the multi-choice question.  ... 
doi:10.1504/ijisd.2009.033083 fatcat:tlgx3bcygzg5pkzefc77d2gnpu

How reliable are annotations via crowdsourcing

Stefanie Nowak, Stefan Rüger
2010 Proceedings of the international conference on Multimedia information retrieval - MIR '10  
crowdsourcing approach.  ...  A subset of the images employed in the latest ImageCLEF Photo Annotation competition was manually annotated by expert annotators and non-experts with Mechanical Turk.  ...  Other work deals with how to verify crowdsourced annotations [4] , how to deal with several noisy labellers [23, 8] and how to balance pricing for crowdsourcing [9] .  ... 
doi:10.1145/1743384.1743478 dblp:conf/mir/NowakR10 fatcat:5cvkjrx2orcrbbhmeuthfxyfbu

Embracing Error to Enable Rapid Crowdsourcing

Ranjay A. Krishna, Kenji Hata, Stephanie Chen, Joshua Kravitz, David A. Shamma, Li Fei-Fei, Michael S. Bernstein
2016 Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems - CHI '16  
Microtask crowdsourcing has enabled dataset advances in social science and machine learning, but existing crowdsourcing schemes are too expensive to scale up with the expanding volume of data.  ...  We evaluate our technique on a breadth of common labeling tasks such as image verification, word similarity, sentiment analysis and topic classification.  ...  Microtasks and Crowdsourcing #chi4good, CHI 2016, San Jose, CA, USA  ... 
doi:10.1145/2858036.2858115 dblp:conf/chi/KrishnaHCKSLB16 fatcat:7s2mfbvgincbrlyzlzucqhmeqa

How to Tell Ancient Signs Apart? Recognizing and Visualizing Maya Glyphs with CNNs

Gülcan Can, Jean-Marc Odobez, Daniel Gatica-Perez
2018 ACM Journal on Computing and Cultural Heritage  
Finally, as a step towards systematic evaluation of these visualizations, we conduct a perceptual crowdsourcing study.  ...  In this context, this paper assesses three different Convolutional Neural Network (CNN) architectures along with three learning approaches to train them for hieroglyph classification, which is a very challenging  ...  As opposed to crowdsourcing studies with every-day objects in natural images, we have the challenge of non-experts not having a predefined concept of glyph categories.  ... 
doi:10.1145/3230670 fatcat:djw46etqgrgadh6ielgtdd662u

TWEET-FID: An Annotated Dataset for Multiple Foodborne Illness Detection Tasks [article]

Ruofan Hu, Dongyu Zhang, Dandan Tao, Thomas Hartvigsen, Hao Feng, Elke Rundensteiner
2022 arXiv   pre-print
TWEET-FID collected from Twitter is annotated with three facets: tweet class, entity type, and slot type, with labels produced by experts as well as by crowdsource workers.  ...  A comprehensive set of results for these tasks leveraging state-of-the-art single- and multi-task deep learning methods on the TWEET-FID dataset are provided.  ...  We thank Mengrui Luo for her assistance in implementing the crowdsourcing interface for this study. References Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. (2018).  ... 
arXiv:2205.10726v1 fatcat:e5njgihxpraxzkcspt6iqkgygm

Service for Crowd-Driven Gathering of Non-Discoverable Knowledge [chapter]

Jim Laredo, Maja Vukovic, Sriram Rajagopal
2012 Lecture Notes in Computer Science  
We developed the system BizRay, instantiating the proposed approach as a general-purpose, self-service Web-based, crowdsourcing service.  ...  In this paper, we describe our novel approach to rapidly design a process solution for a family of business objects, gathering required knowledge through the use of social networking to identify the experts  ...  If the source is not known, the possibility of crowdsourcing with an open call, as described by Howe [12] is an option.  ... 
doi:10.1007/978-3-642-31875-7_39 fatcat:qmmtv2s5xbcbncnefzwnbkszku

A review and experimental analysis of active learning over crowdsourced data

Burcu Sayin, Evgeny Krivosheev, Jie Yang, Andrea Passerini, Fabio Casati
2021 Artificial Intelligence Review  
This paper aims to fill this gap by reviewing the existing active learning approaches and then testing a set of benchmarking ones on crowdsourced datasets.  ...  We provide a comprehensive and systematic survey of the recent research on active learning in the hybrid human–machine classification setting, where crowd workers contribute labels (often noisy) to either  ...  They also propose relabeling the mislabeled items via crowdsourcing.  ... 
doi:10.1007/s10462-021-10021-3 fatcat:gk3iza3vrfeuzeguhqqjduvhyy

Demographic Estimation from Face Images: Human vs. Machine Performance

Hu Han, Charles Otto, Xiaoming Liu, Anil K. Jain
2015 IEEE Transactions on Pattern Analysis and Machine Intelligence  
Given a face image, we first extract demographic informative features via a boosting algorithm, and then employ a hierarchical approach consisting of between-group classification, and withingroup regression  ...  A side-by-side comparison of the demographic estimates from crowdsourced data and the proposed algorithm provides a number of insights into this challenging problem.  ...  Guodong Guo for providing the BIF features of the FG-NET database as a reference. This manuscript benefited from the valuable comments provided by the reviewers.  ... 
doi:10.1109/tpami.2014.2362759 pmid:26357339 fatcat:ngfqhpn5rjf6bfosp7utazrrnu
« Previous Showing results 1 — 15 out of 1,462 results