Filters








4,961 Hits in 5.8 sec

On Adversarial Removal of Hypothesis-only Bias in Natural Language Inference

Yonatan Belinkov, Adam Poliak, Stuart Shieber, Benjamin Van Durme, Alexander Rush
2019 Proceedings of the Eighth Joint Conference on Lexical and Computational Semantics (*  
Popular Natural Language Inference (NLI) datasets have been shown to be tainted by hypothesis-only biases.  ...  We evaluate whether adversarial learning can be used in NLI to encourage models to learn representations free of hypothesis-only biases.  ...  Views and conclusions contained in this publication are those of the authors and should not be interpreted as representing official policies or endorsements of DARPA or the U.S. Government.  ... 
doi:10.18653/v1/s19-1028 dblp:conf/starsem/BelinkovPSDR19 fatcat:gg2l3yzix5du7bpgrlmtpfmsxu

Avoiding the Hypothesis-Only Bias in Natural Language Inference via Ensemble Adversarial Training [article]

Joe Stacey, Pasquale Minervini, Haim Dubossarsky, Sebastian Riedel, Tim Rocktäschel
2021 arXiv   pre-print
Natural Language Inference (NLI) datasets contain annotation artefacts resulting in spurious correlations between the natural language utterances and their respective entailment classes.  ...  In addition, we find that the optimal number of adversarial classifiers depends on the dimensionality of the sentence representations, with larger sentence representations being more difficult to de-bias  ...  Figure 2 : 2 Figure 2: The fall in accuracy of hypothesis-only classifiers when using 1 or 20 adversaries to remove the hypothesis-only bias (compared to a baseline with no adversaries).  ... 
arXiv:2004.07790v5 fatcat:3zwztt6sibfibaguny7forjayq

HypoNLI: Exploring the Artificial Patterns of Hypothesis-only Bias in Natural Language Inference [article]

Tianyu Liu, Xin Zheng, Baobao Chang, Zhifang Sui
2021 arXiv   pre-print
In this work, we manage to derive adversarial examples in terms of the hypothesis-only bias and explore eligible ways to mitigate such bias.  ...  Many recent studies have shown that for models trained on datasets for natural language inference (NLI), it is possible to make correct predictions by merely looking at the hypothesis while completely  ...  This work is supported by the National Science Foundation of China under Grant No. 61751201, No. 61772040, No. 61876004. The corresponding authors of this paper are Baobao Chang and Zhifang Sui.  ... 
arXiv:2003.02756v2 fatcat:me3hcx6v5vfhpdxxpcf5ufltju

Don't Take the Premise for Granted: Mitigating Artifacts in Natural Language Inference

Yonatan Belinkov, Adam Poliak, Stuart Shieber, Benjamin Van Durme, Alexander Rush
2019 Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics  
Natural Language Inference (NLI) datasets often contain hypothesis-only biases-artifacts that allow models to achieve non-trivial performance without learning whether a premise entails a hypothesis.  ...  We evaluate our methods on synthetic and existing NLI datasets by training on datasets containing biases and testing on datasets containing no (or different) hypothesis-only biases.  ...  Views and conclusions contained in this publication are those of the authors and should not be interpreted as representing official policies or endorsements of DARPA or the U.S. Government.  ... 
doi:10.18653/v1/p19-1084 dblp:conf/acl/BelinkovPSDR19 fatcat:mhr2mt4olfd77kb673x4ies6jm

Adversarial Filters of Dataset Biases [article]

Ronan Le Bras, Swabha Swayamdipta, Chandra Bhagavatula, Rowan Zellers, Matthew E. Peters, Ashish Sabharwal, Yejin Choi
2020 arXiv   pre-print
Large neural models have demonstrated human-level performance on language and vision benchmarks, while their performance degrades considerably on adversarial or out-of-distribution samples.  ...  We investigate one recently proposed approach, AFLite, which adversarially filters such dataset biases, as a means to mitigate the prevalent overestimation of machine performance.  ...  Computations on beaker.org were supported in part by credits from Google Cloud.  ... 
arXiv:2002.04108v3 fatcat:qtbwz5g6ajfehlt2x5mue2q4ri

INFOTABS: Inference on Tables as Semi-structured Data [article]

Vivek Gupta, Maitrey Mehta, Pegah Nokhiz, Vivek Srikumar
2020 arXiv   pre-print
In this paper, we observe that semi-structured tabulated text is ubiquitous; understanding them requires not only comprehending the meaning of text fragments, but also implicit relationships between them  ...  Our analysis shows that the semi-structured, multi-domain and heterogeneous nature of the premises admits complex, multi-faceted reasoning.  ...  We acknowledge the support of the support of NSF Grants No. 1822877 and 1801446, and a generous gift from Google.  ... 
arXiv:2005.06117v1 fatcat:p763rr45kbdurh3xtxjygm5rbu

ConjNLI: Natural Language Inference Over Conjunctive Sentences [article]

Swarnadeep Saha, Yixin Nie, Mohit Bansal
2020 arXiv   pre-print
Hence, we introduce ConjNLI, a challenge stress-test for natural language inference over conjunctive sentences, where the premise differs from the hypothesis by conjuncts removed, added, or replaced.  ...  As some initial solutions, we first present an iterative adversarial fine-tuning method that uses synthetically created training data based on boolean and non-boolean heuristics.  ...  The views are those of the authors and not the funding agency.  ... 
arXiv:2010.10418v2 fatcat:4j7kzgvzw5bxden42xf5xbkk7u

Natural Language Inference in Context – Investigating Contextual Reasoning over Long Texts [article]

Hanmeng Liu, Leyang Cui, Jian Liu, Yue Zhang
2020 arXiv   pre-print
Natural language inference (NLI) is a fundamental NLP task, investigating the entailment relationship between two texts. Popular NLI datasets present the task at sentence-level.  ...  Consisting of 8,325 expert-designed "context-hypothesis" pairs with gold labels, ConTRoL is a passage-level NLI dataset with a focus on complex contextual reasoning types such as logical reasoning.  ...  Sentence Persona-chat SciTaiL (Khot, Sabharwal, and Clark 2018a) Natural Language Inference Sentence Science Adversarial NLI (Nie et al. 2019) Natural Language Inference Paragraph Diverse AlphaNLI  ... 
arXiv:2011.04864v1 fatcat:k7ecmhxxibcpjo4euyzapev52a

Do Nuclear Submarines Have Nuclear Captains? A Challenge Dataset for Commonsense Reasoning over Adjectives and Objects

James Mullenbach, Jonathan Gordon, Nanyun Peng, Jonathan May
2019 Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)  
Our findings indicate that, despite the recent successes of large language models on tasks aimed to assess commonsense knowledge, these models do not greatly outperform simple word-level models based on  ...  This provides evidence that the amount of commonsense knowledge encoded in these language models does not extend far beyond that already baked into the word embeddings.  ...  Any opinions, findings, conclusions, or recommendations expressed here are those of the authors and do not necessarily reflect the view of the sponsor.  ... 
doi:10.18653/v1/d19-1625 dblp:conf/emnlp/MullenbachGPM19 fatcat:bdvqejbi5rgc7ndmdlsbxq3icm

VIOLIN: A Large-Scale Dataset for Video-and-Language Inference [article]

Jingzhou Liu, Wenhu Chen, Yu Cheng, Zhe Gan, Licheng Yu, Yiming Yang, Jingjing Liu
2020 arXiv   pre-print
Given a video clip with aligned subtitles as premise, paired with a natural language hypothesis based on the video content, a model needs to infer whether the hypothesis is entailed or contradicted by  ...  ) to in-depth commonsense reasoning (e.g., inferring causal relations of events in the video).  ...  Acknowledgement We would like to thank Yandong Li, Liqun Chen, Shuyang Dai, Linjie Li, Chen Zhu, Jiacheng Xu and Boyi Li for providing useful feedback on the project and their help in collecting and annotating  ... 
arXiv:2003.11618v1 fatcat:zs5m6fonynh27ogqjarxtwrqie

NOPE: A Corpus of Naturally-Occurring Presuppositions in English [article]

Alicia Parrish, Sebastian Schuster, Alex Warstadt, Omar Agha, Soo-Hwan Lee, Zhuoye Zhao, Samuel R. Bowman, Tal Linzen
2021 arXiv   pre-print
Understanding language requires grasping not only the overtly stated content, but also making inferences about things that were left unsaid.  ...  In this work, we introduce the Naturally-Occurring Presuppositions in English (NOPE) Corpus to investigate the context-sensitivity of 10 different types of presupposition triggers and to evaluate machine  ...  A large anno- tated corpus for learning natural language inference. In Proceedings of the 2015 Conference on Empir- ical Methods in Natural Language Processing.  ... 
arXiv:2109.06987v1 fatcat:36x7qdrd6fdqnjxdgh4hjrhioy

Annotation Artifacts in Natural Language Inference Data [article]

Suchin Gururangan, Swabha Swayamdipta, Omer Levy, Roy Schwartz, Samuel R. Bowman, Noah A. Smith
2018 arXiv   pre-print
We show that, in a significant portion of such data, this protocol leaves clues that make it possible to identify the label by looking only at the hypothesis, without observing the premise.  ...  Our findings suggest that the success of natural language inference models to date has been overestimated, and that the task remains a hard open problem.  ...  Acknowledgments This research was supported in part by the DARPA CwC program through ARO (W911NF-15-1-0543) and a hardware gift from NVIDIA Corporation.  ... 
arXiv:1803.02324v2 fatcat:3o57ixhbffgljnlbfhybydco2i

What do Compressed Large Language Models Forget? Robustness Challenges in Model Compression [article]

Mengnan Du, Subhabrata Mukherjee, Yu Cheng, Milad Shokouhi, Xia Hu, Ahmed Hassan Awadallah
2021 arXiv   pre-print
Experimental results on several natural language understanding tasks demonstrate our mitigation framework to improve both the adversarial generalization as well as in-distribution task performance of the  ...  However, there has been no study in analyzing the impact of compression on the generalizability and robustness of these models.  ...  a natural language inference task, which aims to predict whether the relationship between the premise and hypothesis is contradiction, entailment, or neutral.  ... 
arXiv:2110.08419v1 fatcat:3aokxjdva5fwpcynhwtxvcdwyi

Towards Robustifying NLI Models Against Lexical Dataset Biases [article]

Xiang Zhou, Mohit Bansal
2020 arXiv   pre-print
While deep learning models are making fast progress on the task of Natural Language Inference, recent studies have also shown that these models achieve high accuracy by exploiting several dataset biases  ...  Next, we also compare two ways of directly debiasing the model without knowing what the dataset biases are in advance. The first approach aims to remove the label bias at the embedding level.  ...  The views in this article are the authors', not of the funding agency.  ... 
arXiv:2005.04732v2 fatcat:2s4yqi6ykjd6ne7i35now3ovnm

Annotation Artifacts in Natural Language Inference Data

Suchin Gururangan, Swabha Swayamdipta, Omer Levy, Roy Schwartz, Samuel Bowman, Noah A. Smith
2018 Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers)  
We show that, in a significant portion of such data, this protocol leaves clues that make it possible to identify the label by looking only at the hypothesis, without observing the premise.  ...  Our findings suggest that the success of natural language inference models to date has been overestimated, and that the task remains a hard open problem.  ...  Acknowledgments This research was supported in part by the DARPA CwC program through ARO (W911NF-15-1-0543) and a hardware gift from NVIDIA Corporation.  ... 
doi:10.18653/v1/n18-2017 dblp:conf/naacl/GururanganSLSBS18 fatcat:up67r2jwmnawrdsnigw3nuc6h4
« Previous Showing results 1 — 15 out of 4,961 results