A Multilingual Perspective Towards the Evaluation of Attribution Methods in Natural Language Inference [article]

Kerem Zaman, Yonatan Belinkov
2022 arXiv   pre-print
Most evaluations of attribution methods focus on the English language. In this work, we present a multilingual approach for evaluating attribution methods for the Natural Language Inference (NLI) task in terms of plausibility and faithfulness properties. First, we introduce a novel cross-lingual strategy to measure faithfulness based on word alignments, which eliminates the potential downsides of erasure-based evaluations. We then perform a comprehensive evaluation of attribution methods,
more » ... ering different output mechanisms and aggregation methods. Finally, we augment the XNLI dataset with highlight-based explanations, providing a multilingual NLI dataset with highlights, which may support future exNLP studies. Our results show that attribution methods performing best for plausibility and faithfulness are different.
arXiv:2204.05428v1 fatcat:xqvmrzzbwrephpg7qnq4y6ggui