Detecting and Understanding Generalization Barriers for Neural Machine Translation [article]

Guanlin Li, Lemao Liu, Conghui Zhu, Tiejun Zhao, Shuming Shi
2020 arXiv   pre-print
Generalization to unseen instances is our eternal pursuit for all data-driven models. However, for realistic task like machine translation, the traditional approach measuring generalization in an average sense provides poor understanding for the fine-grained generalization ability. As a remedy, this paper attempts to identify and understand generalization barrier words within an unseen input sentence that cause the degradation of fine-grained generalization. We propose a principled definition
more » ... generalization barrier words and a modified version which is tractable in computation. Based on the modified one, we propose three simple methods for barrier detection by the search-aware risk estimation through counterfactual generation. We then conduct extensive analyses on those detected generalization barrier words on both ZhEn NIST benchmarks from various perspectives. Potential usage of the detected barrier words is also discussed.
arXiv:2004.02181v1 fatcat:ik7mnqrg35crhdhn3piw3itira