Seq2Seq2Sentiment: Multimodal Sequence to Sequence Models for Sentiment Analysis [article]

Hai Pham, Thomas Manzini, Paul Pu Liang, Barnabas Poczos
<span title="2018-08-06">2018</span> <i > arXiv </i> &nbsp; <span class="release-stage" >pre-print</span>
Multimodal machine learning is a core research area spanning the language, visual and acoustic modalities. The central challenge in multimodal learning involves learning representations that can process and relate information from multiple modalities. In this paper, we propose two methods for unsupervised learning of joint multimodal representations using sequence to sequence (Seq2Seq) methods: a Seq2Seq Modality Translation Model and a Hierarchical Seq2Seq Modality Translation Model. We also
more &raquo; ... plore multiple different variations on the multimodal inputs and outputs of these seq2seq models. Our experiments on multimodal sentiment analysis using the CMU-MOSI dataset indicate that our methods learn informative multimodal representations that outperform the baselines and achieve improved performance on multimodal sentiment analysis, specifically in the Bimodal case where our model is able to improve F1 Score by twelve points. We also discuss future directions for multimodal Seq2Seq methods.
<span class="external-identifiers"> <a target="_blank" rel="external noopener" href="https://arxiv.org/abs/1807.03915v2">arXiv:1807.03915v2</a> <a target="_blank" rel="external noopener" href="https://fatcat.wiki/release/6zzjqlxfdfa2zdsvhu35xvprle">fatcat:6zzjqlxfdfa2zdsvhu35xvprle</a> </span>
<a target="_blank" rel="noopener" href="https://web.archive.org/web/20200912090436/https://arxiv.org/pdf/1807.03915v2.pdf" title="fulltext PDF download" data-goatcounter-click="serp-fulltext" data-goatcounter-title="serp-fulltext"> <button class="ui simple right pointing dropdown compact black labeled icon button serp-button"> <i class="icon ia-icon"></i> Web Archive [PDF] <div class="menu fulltext-thumbnail"> <img src="https://blobs.fatcat.wiki/thumbnail/pdf/90/97/90978f82022da5ed679ab6d2e1889cd34ca75fa2.180px.jpg" alt="fulltext thumbnail" loading="lazy"> </div> </button> </a> <a target="_blank" rel="external noopener" href="https://arxiv.org/abs/1807.03915v2" title="arxiv.org access"> <button class="ui compact blue labeled icon button serp-button"> <i class="file alternate outline icon"></i> arxiv.org </button> </a>