Deep Spatio-Temporal Features for Multimodal Emotion Recognition

Dung Nguyen, Kien Nguyen, Sridha Sridharan, Afsane Ghasemi, David Dean, Clinton Fookes
2017 2017 IEEE Winter Conference on Applications of Computer Vision (WACV)  
Automatic emotion recognition has attracted great interest and numerous solutions have been proposed, most of which focus either individually on facial expression or acoustic information. While more recent research has considered multimodal approaches, individual modalities are often combined only by simple fusion at the feature and/or decision-level. In this paper, we introduce a novel approach using 3-dimensional convolutional neural networks (C3Ds) to model the spatio-temporal information,
more » ... oral information, cascaded with multimodal deep-belief networks (DBNs) that can represent the audio and video streams. Experiments conducted on the eNTERFACE multimodal emotion database demonstrate that this approach leads to improved multimodal emotion recognition performance and significantly outperforms recent state-of-the-art proposals.
doi:10.1109/wacv.2017.140 dblp:conf/wacv/NguyenNSGDF17 fatcat:b2ectf5ge5gfnioozn5sdrubjy