SceneTrilogy: On Scene Sketches and its Relationship with Text and Photo [article]

Pinaki Nath Chowdhury and Ayan Kumar Bhunia and Tao Xiang and Yi-Zhe Song
2022 arXiv   pre-print
We for the first time extend multi-modal scene understanding to include that of free-hand scene sketches. This uniquely results in a trilogy of scene data modalities (sketch, text, and photo), where each offers unique perspectives for scene understanding, and together enable a series of novel scene-specific applications across discriminative (retrieval) and generative (captioning) tasks. Our key objective is to learn a common three-way embedding space that enables many-to-many modality
more » ... ons (e.g, sketch+text → photo retrieval). We importantly leverage the information bottleneck theory to achieve this goal, where we (i) decouple intra-modality information by minimising the mutual information between modality-specific and modality-agnostic components via a conditional invertible neural network, and (ii) align cross-modalities information by maximising the mutual information between their modality-agnostic components using InfoNCE, with a specific multihead attention mechanism to allow many-to-many modality interactions. We spell out a few insights on the complementarity of each modality for scene understanding, and study for the first time a series of scene-specific applications like joint sketch- and text-based image retrieval, sketch captioning.
arXiv:2204.11964v1 fatcat:r7gaglcvqrhsni6cxqjieppfoy