Internet Archive Scholar

Evaluating Models' Local Decision Boundaries via Contrast Sets [article]

Matt Gardner, Yoav Artzi, Victoria Basmova, Jonathan Berant, Ben Bogin, Sihao Chen, Pradeep Dasigi, Dheeru Dua, Yanai Elazar, Ananth Gottumukkala, Nitish Gupta, Hanna Hajishirzi (+14 others)
<span title="2020-10-01">2020</span> <i > arXiv </i> &nbsp; <span class="release-stage" >pre-print</span>

Preserved Fulltext

Web Archive Capture PDF (2.4 MB)
https://web.archive.org/web/20201008080122/https://arxiv.org/pdf/2004.02709v2.pdf

A copy of this work was available on the public web and has been preserved in the Wayback Machine. The capture dates from 2020; you can also visit <a rel="external noopener" href="https://arxiv.org/pdf/2004.02709v2.pdf">the original URL</a>. The file type is <code>application/pdf</code>.

Other Versions

<span title="2020-04-06">2020</span> <span class="release-stage" >pre-print</span>
<span class="external-identifiers"> <a target="_blank" rel="external noopener" href="https://arxiv.org/abs/2004.02709v1">arXiv:2004.02709v1</a> <a target="_blank" rel="external noopener" href="https://fatcat.wiki/release/2tr2ntk6enf6jmiz6attgag6ha">fatcat:2tr2ntk6enf6jmiz6attgag6ha</a> </span>
Standard test sets for supervised learning evaluate in-distribution generalization. Unfortunately, when a dataset has systematic gaps (e.g., annotation artifacts), these evaluations are misleading: a model can learn simple decision rules that perform well on the test set but do not capture a dataset's intended capabilities. We propose a new annotation paradigm for NLP that helps to close systematic gaps in the test data. In particular, after a dataset is constructed, we recommend that the
more &raquo; ... t authors manually perturb the test instances in small but meaningful ways that (typically) change the gold label, creating contrast sets. Contrast sets provide a local view of a model's decision boundary, which can be used to more accurately evaluate a model's true linguistic capabilities. We demonstrate the efficacy of contrast sets by creating them for 10 diverse NLP datasets (e.g., DROP reading comprehension, UD parsing, IMDb sentiment analysis). Although our contrast sets are not explicitly adversarial, model performance is significantly lower on them than on the original test sets---up to 25\% in some cases. We release our contrast sets as new evaluation benchmarks and encourage future dataset construction efforts to follow similar annotation processes.
<span class="external-identifiers"> <a target="_blank" rel="external noopener" href="https://arxiv.org/abs/2004.02709v2">arXiv:2004.02709v2</a> <a target="_blank" rel="external noopener" href="https://fatcat.wiki/release/zwreyqnxiveyvpktpwazmczfv4">fatcat:zwreyqnxiveyvpktpwazmczfv4</a> </span>
<span class="ui label basic orange small" title="Open Access">Open Access</span> <span class="ui label basic blue small">Multiple Versions</span>
<a target="_blank" rel="noopener" href="https://web.archive.org/web/20201008080122/https://arxiv.org/pdf/2004.02709v2.pdf" title="fulltext PDF download" data-goatcounter-click="serp-fulltext" data-goatcounter-title="serp-fulltext"> <button class="ui simple right pointing dropdown compact black labeled icon button serp-button"> <i class="icon ia-icon"></i> Web Archive [PDF] </button> </a> <a target="_blank" rel="external noopener" href="https://arxiv.org/abs/2004.02709v2" title="arxiv.org access"> <button class="ui compact blue labeled icon button serp-button"> <i class="file alternate outline icon"></i> arxiv.org </button> </a>
Citation

Matt Gardner, Yoav Artzi, Victoria Basmova, Jonathan Berant, Ben Bogin, Sihao Chen, Pradeep Dasigi, Dheeru Dua, Yanai Elazar, Ananth Gottumukkala, Nitish Gupta, Hanna Hajishirzi, Gabriel Ilharco, Daniel Khashabi, Kevin Lin, Jiangming Liu, Nelson F. Liu, Phoebe Mulcaire, Qiang Ning, Sameer Singh, Noah A. Smith, Sanjay Subramanian, Reut Tsarfaty, Eric Wallace, Ally Zhang, Ben Zhou. "Evaluating Models' Local Decision Boundaries via Contrast Sets." arXiv (2020)