A new approach to integrate audio and visual features of speech

H. Pan, Zhi-Pei Liang, T.S. Huang
2000 IEEE International Conference on Multimedia and Expo. ICME2000. Proceedings. Latest Advances in the Fast Changing World of Multimedia (Cat. No.00TH8532)  
This paper presents a novel fused-hidden Markov model (fused-HMM) to integrate the audio and visual features of speech. In this model, audio and visual HMMs built individually are fused together using a general probabilistic fusion method, which is optimal in the maximum entropy sense. Specifically, the fusion method uses the dependencies between the audio hidden states and the visual observations to infer the dependencies between audio and video. The learning and inference algorithms described
more » ... lgorithms described in this paper can handle audio and video features with different data rates and duration. In speaker verification experiments, the results show that the proposed method significantly reduces the recognition error rate as compared to unimodal HMMs and other simpler fusion methods.
doi:10.1109/icme.2000.871551 dblp:conf/icmcs/PanLH00 fatcat:jmuvfmoxvfasbi4qdqqixxwvoq