All-in-One Image-Grounded Conversational Agents [article]

Da Ju, Kurt Shuster, Y-Lan Boureau, Jason Weston
2020 arXiv   pre-print
As single-task accuracy on individual language and image tasks has improved substantially in the last few years, the long-term goal of a generally skilled agent that can both see and talk becomes more feasible to explore. In this work, we focus on leveraging individual language and image tasks, along with resources that incorporate both vision and language towards that objective. We design an architecture that combines state-of-the-art Transformer and ResNeXt modules fed into a novel attentive
more » ... ultimodal module to produce a combined model trained on many tasks. We provide a thorough analysis of the components of the model, and transfer performance when training on one, some, or all of the tasks. Our final models provide a single system that obtains good results on all vision and language tasks considered, and improves the state-of-the-art in image-grounded conversational applications.
arXiv:1912.12394v2 fatcat:heqeepgwsbbdfgdixum3cwb33y