It's Time for Artistic Correspondence in Music and Video [article]

Didac Suris, Carl Vondrick, Bryan Russell, Justin Salamon
2022 arXiv   pre-print
We present an approach for recommending a music track for a given video, and vice versa, based on both their temporal alignment and their correspondence at an artistic level. We propose a self-supervised approach that learns this correspondence directly from data, without any need of human annotations. In order to capture the high-level concepts that are required to solve the task, we propose modeling the long-term temporal context of both the video and the music signals, using Transformer
more » ... rks for each modality. Experiments show that this approach strongly outperforms alternatives that do not exploit the temporal context. The combination of our contributions improve retrieval accuracy up to 10x over prior state of the art. This strong improvement allows us to introduce a wide range of analyses and applications. For instance, we can condition music retrieval based on visually defined attributes.
arXiv:2206.07148v1 fatcat:5kiflxr34vayxbcmcnjlg5dniu