Filters








1,222 Hits in 7.0 sec

Truth inference in crowdsourcing

Yudian Zheng, Guoliang Li, Yuanbing Li, Caihua Shan, Reynold Cheng
2017 Proceedings of the VLDB Endowment  
However, due to the openness of crowdsourcing, workers may yield low-quality answers, and a redundancy-based method is widely employed, which first assigns each task to multiple workers and then infers  ...  A fundamental problem in this method is Truth Inference, which decides how to effectively infer the truth.  ...  We summarize a framework (Algorithm 1) and analyze the task types, task models, worker models and inference techniques in these methods.  ... 
doi:10.14778/3055540.3055547 fatcat:wijrseqew5dpleeynxk3f2eapq

Achieving Budget-optimality with Adaptive Schemes in Crowdsourcing [article]

Ashish Khetan, Sewoong Oh
2017 arXiv   pre-print
Under this generalized Dawid-Skene model, we characterize the fundamental trade-off between budget and accuracy. We introduce a novel adaptive scheme that matches this fundamental limit.  ...  In particular, we ask whether adaptive task assignment schemes lead to a more efficient trade-off between the accuracy and the budget.  ...  Acknowledgements This work is supported by NSF SaTC award CNS-1527754, NSF CISE award CCF-1553452, NSF CISE award CCF-1705007 and GOOGLE Faculty Research Award.  ... 
arXiv:1602.03481v3 fatcat:qorrngjnffe7fjkd2lnnjbvwfe

Budget-Optimal Task Allocation for Reliable Crowdsourcing Systems [article]

David R. Karger and Sewoong Oh and Devavrat Shah
2013 arXiv   pre-print
We give a new algorithm for deciding which tasks to assign to which workers and for inferring correct answers from the workers' answers.  ...  In this paper, we consider a general model of such crowdsourcing tasks and pose the problem of minimizing the total price (i.e., number of task assignments) that must be paid to achieve a target overall  ...  Most of the models studied in the crowdsourcing literature can be reduced to a special case of this model.  ... 
arXiv:1110.3564v4 fatcat:h3nptybwuvggrhpekfxwohhu4q

Optimal Inference in Crowdsourced Classification via Belief Propagation [article]

Jungseul Ok, Sewoong Oh, Jinwoo Shin, Yung Yi
2017 arXiv   pre-print
Crowdsourcing systems are popular for solving large-scale labelling tasks with low-paid workers.  ...  We close this gap by introducing a tighter lower bound on the fundamental limit and proving that Belief Propagation (BP) exactly matches this lower bound.  ...  Acknowledgment This work is supported by NSF SaTC award CNS-1527754, and NSF CISE award CCF-1553452.  ... 
arXiv:1602.03619v4 fatcat:h6sfpqnnl5anxjjwlsgn7ikpuy

Budget-Optimal Task Allocation for Reliable Crowdsourcing Systems

David R. Karger, Sewoong Oh, Devavrat Shah
2014 Operations Research  
We give a new algorithm for deciding which tasks to assign to which workers and for inferring correct answers from the workers' answers.  ...  In this paper, we consider a general model of such crowdsourcing tasks and pose the problem of minimizing the total price (i.e., number of task assignments) that must be paid to achieve a target overall  ...  The crowdsourcing model we consider in this paper is a special case of these models, and we discuss their relationship in Section 2.7.  ... 
doi:10.1287/opre.2013.1235 fatcat:ozoasgl7tvfpbfctwpmbjscuhy

Crowdsourced Entity Alignment: A Decision Theory Based Approach [chapter]

Yan Zhuang, Guoliang Li, Jianhua Feng
2017 Lecture Notes in Computer Science  
Crowdsourcing is a new computation paradigm that utilizes the wisdom of the crowd to solve problems which are difficult for computers (e.g., image annotation and entity alignment).  ...  In crowdsourced entity alignment tasks, there are usually large numbers of candidate pairs to be verified by the crowd workers, and each pair will be assigned to multiple workers to achieve high quality  ...  Crowdsourcing Model There are many crowdsourcing platforms, such as AMT and CrowdFlower, which provide facilities for asking online workers to complete the alignment tasks.  ... 
doi:10.1007/978-3-319-68786-5_2 fatcat:ui55qrip4zalrpizpd54uqpkmu

Crowdsourced Data Management: A Survey

Guoliang Li, Jiannan Wang, Yudian Zheng, Michael J. Franklin
2016 IEEE Transactions on Knowledge and Data Engineering  
., the crowd) to apply human computation for such tasks. Thus, crowdsourced data management has become an area of increasing interest in research and industry.  ...  In this paper, we survey and synthesize a wide spectrum of existing studies on crowdsourced data management.  ...  [73] model workers and tasks using a directed graph, and apply variational inference methods to iteratively derive workers' models. 1.  ... 
doi:10.1109/tkde.2016.2535242 fatcat:sit3comyvra4djqkl4xre3kj24

Fundamental Limits of Budget-Fidelity Trade-off in Label Crowdsourcing [article]

Farshad Lahouti, Babak Hassibi
2016 arXiv   pre-print
The crowdsourcer then collects and analyzes the results for inference and serving the purpose of the project.  ...  In CS, a taskmaster typically breaks down the project into small batches of tasks and assigns them to so-called workers with imperfect skill levels.  ...  The model, as depicted in Figure 1a , then enables the analysis of the fundamental performance limits of crowdsourcing.  ... 
arXiv:1608.07328v1 fatcat:gri6yyofkjcitdafzvl6akedrq

Budget-optimal crowdsourcing using low-rank matrix approximations

David R. Karger, Sewoong Oh, Devavrat Shah
2011 2011 49th Annual Allerton Conference on Communication, Control, and Computing (Allerton)  
We give a new algorithm for deciding which tasks to assign to which workers and for inferring correct answers from the workers' answers.  ...  In this paper, we consider a model of such crowdsourcing tasks and pose the problem of minimizing the total price (i.e., number of task assignments) that must be paid to achieve a target overall reliability  ...  Most of the crowdsourcing models introduced so far can be reduced to a special case of this model.  ... 
doi:10.1109/allerton.2011.6120180 dblp:conf/allerton/KargerOS11 fatcat:wdaqbc52qnf4lcubv2hbnyogeu

Reply & Supply: Efficient crowdsourcing when workers do more than answer questions

Thomas C. McAndrew, Elizaveta A. Guseva, James P. Bagrow, Zhong-Ke Gao
2017 PLoS ONE  
Crowdsourcing works by distributing many small tasks to large numbers of workers, yet the true potential of crowdsourcing lies in workers doing more than performing simple tasks---they can apply their  ...  By modeling question sets as networks of interrelated questions, we introduce algorithms to help curtail the growth bias by efficiently distributing workers between exploring new questions and addressing  ...  Wagy and J. Bongard for useful comments and gratefully acknowledge the resources provided by the Vermont Advanced Computing Core.  ... 
doi:10.1371/journal.pone.0182662 pmid:28806413 pmcid:PMC5555646 fatcat:4tqv6xdpxbfwjfphsxptrd6a6y

Power of Bonus in Pricing for Crowdsourcing [article]

Suho Shin, Hoyong Choi, Yung Yi, Jungseul Ok
2021 arXiv   pre-print
We consider a simple form of pricing for a crowdsourcing system, where pricing policy is published a priori, and workers then decide their task acceptance.  ...  With the goal of designing efficient and simple pricing rules, we study the impact of the following two design features in pricing policies: (i) personalization tailoring policy worker-by-worker and (ii  ...  Model and Problem Formulation System Model We consider a set N = {1, 2, . . . , n} of workers, where n is the number of workers who are available for a target task requested by the requester.  ... 
arXiv:1804.03178v4 fatcat:6i4u3lgqonh7bbcvvw2knxhaqu

Towards Crowdsourced Training of Large Neural Networks using Decentralized Mixture-of-Experts [article]

Max Ryabinin, Anton Gusev
2020 arXiv   pre-print
For instance, the cluster used to train GPT-3 costs over $250 million. As a result, most researchers cannot afford to train state of the art models and contribute to their development.  ...  Hypothetically, a researcher could crowdsource the training of large neural networks with thousands of regular PCs provided by volunteers.  ...  Also, we are grateful to Vladimir Aliev for proofreading and helping us improve the presentation of the paper.  ... 
arXiv:2002.04013v3 fatcat:c5u36uzdenannmgtj6m3bpq2sm

ZebraLancer: Decentralized Crowdsourcing of Human Knowledge atop Open Blockchain [article]

Yuan Lu, Qiang Tang, Guiling Wang
2020 arXiv   pre-print
Finally, we implement our protocol for a common image annotation task and deploy it in a test net of Ethereum.  ...  We design and implement the first private and anonymous decentralized crowdsourcing system ZebraLancer, and overcome two fundamental challenges of decentralizing crowdsourcing, i.e., data leakage and identity  ...  When the requester R has a crowdsourcing task, she generates a fresh blockchain account address α R , and a key pair (epk, esk) (which will be used for workers to encrypt submissions) for this task only  ... 
arXiv:1803.01256v5 fatcat:fgynlvgvxvca3agaoqa42m6azm

Crowdsourced Data Management: Industry and Academic Perspectives

Adam Marcus, Aditya Parameswaran
2015 Foundations and Trends in Databases  
Crowdsourcing and human computation enable organizations to accomplish tasks that are currently not possible for fully automated techniques to complete, or require more flexibility and scalability than  ...  In the area of data processing, companies have benefited from crowd workers on platforms such as Amazon's Mechanical Turk or Upwork to complete tasks as varied as content moderation, web content extraction  ...  Most crowdsourced workflows implicitly or explicitly compensate workers for the time they spend on a task, and measuring the efficiency with which a worker completes a task is important to maintaining  ... 
doi:10.1561/1900000044 fatcat:rva7dinutnbnlj2hvm4j2hmhge

Distribution-Aware Crowdsourced Entity Collection

Ju Fan, Zhewei Wei, Dongxiang Zhang, Jingru Yang, Xiaoyong Du
2016 IEEE Transactions on Knowledge and Data Engineering  
The problem of crowdsourced entity collection solicits people (a.k.a. workers) to complete missing data in a database and has witnessed many applications in knowledge base completion and enterprise data  ...  a set of entities via crowdsourcing and minimize the difference of the entity distribution from the expected distribution.  ...  Crowd Model for Crowdsourced Entity Collection We employ crowdsourcing for entity collection.  ... 
doi:10.1109/tkde.2016.2611509 fatcat:4c4zac6qsfacdi5kgggqndk4ge
« Previous Showing results 1 — 15 out of 1,222 results