Internet Archive Scholar

Easy, Reproducible and Quality-Controlled Data Collection with Crowdaq [article]

Qiang Ning, Hao Wu, Pradeep Dasigi, Dheeru Dua, Matt Gardner, Robert L. Logan IV, Ana Marasovic, Zhen Nie
<span title="2020-10-06">2020</span> <i > arXiv </i> &nbsp; <span class="release-stage" >pre-print</span>

Preserved Fulltext

fulltext thumbnail
Web Archive Capture PDF (3.0 MB)
https://web.archive.org/web/20201120194657/https://arxiv.org/pdf/2010.06694v1.pdf

A copy of this work was available on the public web and has been preserved in the Wayback Machine. The capture dates from 2020; you can also visit <a rel="external noopener" href="https://arxiv.org/pdf/2010.06694v1.pdf">the original URL</a>. The file type is <code>application/pdf</code>.

High-quality and large-scale data are key to success for AI systems. However, large-scale data annotation efforts are often confronted with a set of common challenges: (1) designing a user-friendly annotation interface; (2) training enough annotators efficiently; and (3) reproducibility. To address these problems, we introduce Crowdaq, an open-source platform that standardizes the data collection pipeline with customizable user-interface components, automated annotator qualification, and saved
more &raquo; ... ipelines in a re-usable format. We show that Crowdaq simplifies data annotation significantly on a diverse set of data collection use cases and we hope it will be a convenient tool for the community.
<span class="external-identifiers"> <a target="_blank" rel="external noopener" href="https://arxiv.org/abs/2010.06694v1">arXiv:2010.06694v1</a> <a target="_blank" rel="external noopener" href="https://fatcat.wiki/release/5jnkjtuz4vehzea7dmntepwnja">fatcat:5jnkjtuz4vehzea7dmntepwnja</a> </span>
<a target="_blank" rel="noopener" href="https://web.archive.org/web/20201120194657/https://arxiv.org/pdf/2010.06694v1.pdf" title="fulltext PDF download" data-goatcounter-click="serp-fulltext" data-goatcounter-title="serp-fulltext"> <button class="ui simple right pointing dropdown compact black labeled icon button serp-button"> <i class="icon ia-icon"></i> Web Archive [PDF] <div class="menu fulltext-thumbnail"> <img src="https://blobs.fatcat.wiki/thumbnail/pdf/a0/c5/a0c52a5cca698636b5a516b24f824d23f506f6e8.180px.jpg" alt="fulltext thumbnail" loading="lazy"> </div> </button> </a> <a target="_blank" rel="external noopener" href="https://arxiv.org/abs/2010.06694v1" title="arxiv.org access"> <button class="ui compact blue labeled icon button serp-button"> <i class="file alternate outline icon"></i> arxiv.org </button> </a>
Citation

Qiang Ning, Hao Wu, Pradeep Dasigi, Dheeru Dua, Matt Gardner, Robert L. Logan IV, Ana Marasovic, Zhen Nie. "Easy, Reproducible and Quality-Controlled Data Collection with Crowdaq." arXiv (2020)