Computer Vision and Natural Language Processing

Peratham Wiriyathammabhum, Douglas Summers-Stay, Cornelia Fermüller, Yiannis Aloimonos
2016 ACM Computing Surveys  
Integrating computer vision and natural language processing is a novel interdisciplinary field that has received a lot of attention recently. In this survey, we provide a comprehensive introduction of the integration of computer vision and natural language processing in multimedia and robotics applications with more than 200 key references. The tasks that we survey include visual attributes, image captioning, video captioning, visual question answering, visual retrieval, human-robot
more » ... robotic actions, and robot navigation. We also emphasize strategies to integrate computer vision and natural language processing models as a unified theme of distributional semantics. We make an analog of distributional semantics in computer vision and natural language processing as image embedding and word embedding, respectively. We also present a unified view for the field and propose possible future directions.
doi:10.1145/3009906 fatcat:bdgaeoz4w5djhd5spab4lrc4au