Multimodal Hierarchical Reinforcement Learning Policy for Task-Oriented Visual Dialog [article]

Jiaping Zhang, Tiancheng Zhao, Zhou Yu
<span title="2018-05-08">2018</span> <i > arXiv </i> &nbsp; <span class="release-stage" >pre-print</span>
Creating an intelligent conversational system that understands vision and language is one of the ultimate goals in Artificial Intelligence (AI) winograd1972understanding. Extensive research has focused on vision-to-language generation, however, limited research has touched on combining these two modalities in a goal-driven dialog context. We propose a multimodal hierarchical reinforcement learning framework that dynamically integrates vision and language for task-oriented visual dialog. The
more &raquo; ... ework jointly learns the multimodal dialog state representation and the hierarchical dialog policy to improve both dialog task success and efficiency. We also propose a new technique, state adaptation, to integrate context awareness in the dialog state representation. We evaluate the proposed framework and the state adaptation technique in an image guessing game and achieve promising results.
<span class="external-identifiers"> <a target="_blank" rel="external noopener" href="https://arxiv.org/abs/1805.03257v1">arXiv:1805.03257v1</a> <a target="_blank" rel="external noopener" href="https://fatcat.wiki/release/dg5npzmbcrcizbwonzf75eodfy">fatcat:dg5npzmbcrcizbwonzf75eodfy</a> </span>
<a target="_blank" rel="noopener" href="https://web.archive.org/web/20191020193913/https://arxiv.org/pdf/1805.03257v1.pdf" title="fulltext PDF download" data-goatcounter-click="serp-fulltext" data-goatcounter-title="serp-fulltext"> <button class="ui simple right pointing dropdown compact black labeled icon button serp-button"> <i class="icon ia-icon"></i> Web Archive [PDF] <div class="menu fulltext-thumbnail"> <img src="https://blobs.fatcat.wiki/thumbnail/pdf/2c/1f/2c1f9000475b5778223487f605487e8f3914968b.180px.jpg" alt="fulltext thumbnail" loading="lazy"> </div> </button> </a> <a target="_blank" rel="external noopener" href="https://arxiv.org/abs/1805.03257v1" title="arxiv.org access"> <button class="ui compact blue labeled icon button serp-button"> <i class="file alternate outline icon"></i> arxiv.org </button> </a>