Training an adaptive dialogue policy for interactive learning of visually grounded word meanings

Yanchao Yu, Arash Eshghi, Oliver Lemon
2016 Proceedings of the 17th Annual Meeting of the Special Interest Group on Discourse and Dialogue  
We present a multi-modal dialogue system for interactive learning of perceptually grounded word meanings from a human tutor. The system integrates an incremental, semantic parsing/generation framework - Dynamic Syntax and Type Theory with Records (DS-TTR) - with a set of visual classifiers that are learned throughout the interaction and which ground the meaning representations that it produces. We use this system in interaction with a simulated human tutor to study the effects of different
more » ... gue policies and capabilities on the accuracy of learned meanings, learning rates, and efforts/costs to the tutor. We show that the overall performance of the learning agent is affected by (1) who takes initiative in the dialogues; (2) the ability to express/use their confidence level about visual attributes; and (3) the ability to process elliptical and incrementally constructed dialogue turns. Ultimately, we train an adaptive dialogue policy which optimises the trade-off between classifier accuracy and tutoring costs.
doi:10.18653/v1/w16-3643 dblp:conf/sigdial/YuEL16 fatcat:3h34ypvjordglfa3lg4fzubkay