Do As I Do: Transferring Human Motion and Appearance between Monocular Videos with Spatial and Temporal Constraints [article]

Thiago L. Gomes and Renato Martins and João Ferreira and Erickson R. Nascimento
<span title="2020-01-21">2020</span> <i > arXiv </i> &nbsp; <span class="release-stage" >pre-print</span>
Creating plausible virtual actors from images of real actors remains one of the key challenges in computer vision and computer graphics. Marker-less human motion estimation and shape modeling from images in the wild bring this challenge to the fore. Although the recent advances on view synthesis and image-to-image translation, currently available formulations are limited to transfer solely style and do not take into account the character's motion and shape, which are by nature intermingled to
more &raquo; ... oduce plausible human forms. In this paper, we propose a unifying formulation for transferring appearance and retargeting human motion from monocular videos that regards all these aspects. Our method synthesizes new videos of people in a different context where they were initially recorded. Differently from recent appearance transferring methods, our approach takes into account body shape, appearance, and motion constraints. The evaluation is performed with several experiments using publicly available real videos containing hard conditions. Our method is able to transfer both human motion and appearance outperforming state-of-the-art methods, while preserving specific features of the motion that must be maintained (e.g., feet touching the floor, hands touching a particular object) and holding the best visual quality and appearance metrics such as Structural Similarity (SSIM) and Learned Perceptual Image Patch Similarity (LPIPS).
<span class="external-identifiers"> <a target="_blank" rel="external noopener" href="https://arxiv.org/abs/2001.02606v2">arXiv:2001.02606v2</a> <a target="_blank" rel="external noopener" href="https://fatcat.wiki/release/m2cposgyqneqvnxsxznrtr4akm">fatcat:m2cposgyqneqvnxsxznrtr4akm</a> </span>
<a target="_blank" rel="noopener" href="https://web.archive.org/web/20200321070501/https://arxiv.org/pdf/2001.02606v2.pdf" title="fulltext PDF download" data-goatcounter-click="serp-fulltext" data-goatcounter-title="serp-fulltext"> <button class="ui simple right pointing dropdown compact black labeled icon button serp-button"> <i class="icon ia-icon"></i> Web Archive [PDF] </button> </a> <a target="_blank" rel="external noopener" href="https://arxiv.org/abs/2001.02606v2" title="arxiv.org access"> <button class="ui compact blue labeled icon button serp-button"> <i class="file alternate outline icon"></i> arxiv.org </button> </a>