To React or not to React: End-to-End Visual Pose Forecasting for Personalized Avatar during Dyadic Conversations [article]

Chaitanya Ahuja, Shugao Ma, Louis-Philippe Morency, Yaser Sheikh
<span title="2019-10-05">2019</span> <i > arXiv </i> &nbsp; <span class="release-stage" >pre-print</span>
Non verbal behaviours such as gestures, facial expressions, body posture, and para-linguistic cues have been shown to complement or clarify verbal messages. Hence to improve telepresence, in form of an avatar, it is important to model these behaviours, especially in dyadic interactions. Creating such personalized avatars not only requires to model intrapersonal dynamics between a avatar's speech and their body pose, but it also needs to model interpersonal dynamics with the interlocutor present
more &raquo; ... in the conversation. In this paper, we introduce a neural architecture named Dyadic Residual-Attention Model (DRAM), which integrates intrapersonal (monadic) and interpersonal (dyadic) dynamics using selective attention to generate sequences of body pose conditioned on audio and body pose of the interlocutor and audio of the human operating the avatar. We evaluate our proposed model on dyadic conversational data consisting of pose and audio of both participants, confirming the importance of adaptive attention between monadic and dyadic dynamics when predicting avatar pose. We also conduct a user study to analyze judgments of human observers. Our results confirm that the generated body pose is more natural, models intrapersonal dynamics and interpersonal dynamics better than non-adaptive monadic/dyadic models.
<span class="external-identifiers"> <a target="_blank" rel="external noopener" href="https://arxiv.org/abs/1910.02181v1">arXiv:1910.02181v1</a> <a target="_blank" rel="external noopener" href="https://fatcat.wiki/release/4ozdq6rhkja4pnvlytoe3tpzia">fatcat:4ozdq6rhkja4pnvlytoe3tpzia</a> </span>
<a target="_blank" rel="noopener" href="https://web.archive.org/web/20200828223729/https://arxiv.org/pdf/1910.02181v1.pdf" title="fulltext PDF download" data-goatcounter-click="serp-fulltext" data-goatcounter-title="serp-fulltext"> <button class="ui simple right pointing dropdown compact black labeled icon button serp-button"> <i class="icon ia-icon"></i> Web Archive [PDF] <div class="menu fulltext-thumbnail"> <img src="https://blobs.fatcat.wiki/thumbnail/pdf/af/92/af92dd61340808f3008a84ae57803bb4aa57d03b.180px.jpg" alt="fulltext thumbnail" loading="lazy"> </div> </button> </a> <a target="_blank" rel="external noopener" href="https://arxiv.org/abs/1910.02181v1" title="arxiv.org access"> <button class="ui compact blue labeled icon button serp-button"> <i class="file alternate outline icon"></i> arxiv.org </button> </a>