Regularizing RNNs for Caption Generation by Reconstructing the Past with the Present

Xinpeng Chen, Lin Ma, Wenhao Jiang, Jian Yao, Wei Liu
2018 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition  
Recently, caption generation with an encoder-decoder framework has been extensively studied and applied in different domains, such as image captioning, code captioning, and so on. In this paper, we propose a novel architecture, namely Auto-Reconstructor Network (ARNet), which, coupling with the conventional encoder-decoder framework, works in an end-to-end fashion to generate captions. AR-Net aims at reconstructing the previous hidden state with the present one, besides behaving as the
more » ... endent transition operator. Therefore, ARNet encourages the current hidden state to embed more information from the previous one, which can help regularize the transition dynamics of recurrent neural networks (RNNs). Extensive experimental results show that our proposed ARNet boosts the performance over the existing encoder-decoder models on both image captioning and source code captioning tasks. Additionally, ARNet remarkably reduces the discrepancy between training and inference processes for caption generation. Furthermore, the performance on permuted sequential MNIST demonstrates that ARNet can effectively regularize RNN, especially on modeling long-term dependencies. Our code is available at: https://github.com/ chenxinpeng/ARNet.
doi:10.1109/cvpr.2018.00834 dblp:conf/cvpr/Chen0JY018 fatcat:7mporxrrtfa45jckbiv7oujp2q