Depth Infused Binaural Audio Generation using Hierarchical Cross-Modal Attention [article]

Kranti Kumar Parida, Siddharth Srivastava, Neeraj Matiyali, Gaurav Sharma
<span title="2021-08-10">2021</span> <i > arXiv </i> &nbsp; <span class="release-stage" >pre-print</span>
Binaural audio gives the listener the feeling of being in the recording place and enhances the immersive experience if coupled with AR/VR. But the problem with binaural audio recording is that it requires a specialized setup which is not possible to fabricate within handheld devices as compared to traditional mono audio that can be recorded with a single microphone. In order to overcome this drawback, prior works have tried to uplift the mono recorded audio to binaural audio as a post
more &raquo; ... step conditioning on the visual input. But all the prior approaches missed other most important information required for the task, i.e. distance of different sound producing objects from the recording setup. In this work, we argue that the depth map of the scene can act as a proxy for encoding distance information of objects in the scene and show that adding depth features along with image features improves the performance both qualitatively and quantitatively. We propose a novel encoder-decoder architecture, where we use a hierarchical attention mechanism to encode the image and depth feature extracted from individual transformer backbone, with audio features at each layer of the decoder.
<span class="external-identifiers"> <a target="_blank" rel="external noopener" href="https://arxiv.org/abs/2108.04906v1">arXiv:2108.04906v1</a> <a target="_blank" rel="external noopener" href="https://fatcat.wiki/release/bo6kd7yj7ffeppctjuob2ph7tu">fatcat:bo6kd7yj7ffeppctjuob2ph7tu</a> </span>
<a target="_blank" rel="noopener" href="https://web.archive.org/web/20210814102228/https://arxiv.org/pdf/2108.04906v1.pdf" title="fulltext PDF download" data-goatcounter-click="serp-fulltext" data-goatcounter-title="serp-fulltext"> <button class="ui simple right pointing dropdown compact black labeled icon button serp-button"> <i class="icon ia-icon"></i> Web Archive [PDF] <div class="menu fulltext-thumbnail"> <img src="https://blobs.fatcat.wiki/thumbnail/pdf/7d/c2/7dc2c9066308f1b4dfe5fde9d4e84cfcf98fc095.180px.jpg" alt="fulltext thumbnail" loading="lazy"> </div> </button> </a> <a target="_blank" rel="external noopener" href="https://arxiv.org/abs/2108.04906v1" title="arxiv.org access"> <button class="ui compact blue labeled icon button serp-button"> <i class="file alternate outline icon"></i> arxiv.org </button> </a>