Visual Agreement Regularized Training for Multi-Modal Machine Translation [article]

Pengcheng Yang, Boxing Chen, Pei Zhang, Xu Sun
<span title="2019-12-27">2019</span> <i > arXiv </i> &nbsp; <span class="release-stage" >pre-print</span>
Multi-modal machine translation aims at translating the source sentence into a different language in the presence of the paired image. Previous work suggests that additional visual information only provides dispensable help to translation, which is needed in several very special cases such as translating ambiguous words. To make better use of visual information, this work presents visual agreement regularized training. The proposed approach jointly trains the source-to-target and
more &raquo; ... e translation models and encourages them to share the same focus on the visual information when generating semantically equivalent visual words (e.g. "ball" in English and "ballon" in French). Besides, a simple yet effective multi-head co-attention model is also introduced to capture interactions between visual and textual features. The results show that our approaches can outperform competitive baselines by a large margin on the Multi30k dataset. Further analysis demonstrates that the proposed regularized training can effectively improve the agreement of attention on the image, leading to better use of visual information.
<span class="external-identifiers"> <a target="_blank" rel="external noopener" href="https://arxiv.org/abs/1912.12014v1">arXiv:1912.12014v1</a> <a target="_blank" rel="external noopener" href="https://fatcat.wiki/release/7ifibrznxrfujph6trfvurbo5q">fatcat:7ifibrznxrfujph6trfvurbo5q</a> </span>
<a target="_blank" rel="noopener" href="https://web.archive.org/web/20200321040518/https://arxiv.org/pdf/1912.12014v1.pdf" title="fulltext PDF download" data-goatcounter-click="serp-fulltext" data-goatcounter-title="serp-fulltext"> <button class="ui simple right pointing dropdown compact black labeled icon button serp-button"> <i class="icon ia-icon"></i> Web Archive [PDF] </button> </a> <a target="_blank" rel="external noopener" href="https://arxiv.org/abs/1912.12014v1" title="arxiv.org access"> <button class="ui compact blue labeled icon button serp-button"> <i class="file alternate outline icon"></i> arxiv.org </button> </a>