Towards rich multimodal behavior in spoken dialogues with embodied agents

Samer Al Moubayed
2013 2013 IEEE 4th International Conference on Cognitive Infocommunications (CogInfoCom)  
Spoken dialogue frameworks have traditionally been designed to handle a single stream of datathe speech signal. Research on human-human communication has been providing large evidence and quantifying the effects and the importance of a multitude of other multimodal nonverbal signals that people use in their communication, that shape and regulate their interaction. Driven by findings from multimodal human spoken interaction, and the advancements of capture devices and robotics and animation
more » ... ologies, new possibilities are rising for the development of multimodal human-machine interaction that is more affective, social, and engaging. In such face-to-face interaction scenarios, dialogue systems can have a large set of signals at their disposal to infer context and enhance and regulate the interaction through the generation of verbal and nonverbal facial signals. This paper summarizes several design decision, and experiments that we have followed in attempts to build rich and fluent multimodal interactive systems using a newly developed hybrid robotic head called Furhat, and discuss issues and challenges that this effort is facing.
doi:10.1109/coginfocom.2013.6719212 fatcat:snabtnoxhbd4hcy6u73dzti76e