Transcription-Enriched Joint Embeddings for Spoken Descriptions of Images and Videos [article]

Benet Oriol, Jordi Luque, Ferran Diego, Xavier Giro-i-Nieto
2020 arXiv   pre-print
In this work, we propose an effective approach for training unique embedding representations by combining three simultaneous modalities: image and spoken and textual narratives. The proposed methodology departs from a baseline system that spawns a embedding space trained with only spoken narratives and image cues. Our experiments on the EPIC-Kitchen and Places Audio Caption datasets show that introducing the human-generated textual transcriptions of the spoken narratives helps to the training
more » ... ocedure yielding to get better embedding representations. The triad speech, image and words allows for a better estimate of the point embedding and show an improving of the performance within tasks like image and speech retrieval, even when text third modality, text, is not present in the task.
arXiv:2006.00785v1 fatcat:yee3epxdozhgdalg3tg3d7zxwq