Real-time lip-tracking for lipreading

Rainer Stiefelhagen, Uwe Meier, Jie Yang
This paper presents a new approach to lip tracking for lipreading. Instead of only tracking features on lips, we propose to track lips along with other facial features such as pupils and nostril. In the new approach, the face is rst located in an image using a stochastic skin-color model, the eyes, lip-corners and nostrils are then located and tracked inside the facial region. The new approach can e ectively improve the robustness of lip-tracking and simplify automatic detection and recovery of
more » ... ion and recovery of tracking failure. The feasibility of the proposed approach has been demonstrated by implementation of a lip tracking system. The system has been tested by a database that contains 900 image sequences of di erent speakers spelling words. The system has successfully extract lip regions from the image sequences to obtain training data for the audio-visual speech recognition system. The system has been also applied to extract the lip region in real-time from live video images to obtain the visual input for an audio-visual speech recognition system. On test sequences we have achieved a reduction of the number of frames with tracking failures by a factor of two using detection and prediction of outliers in the set of found features.
doi:10.5445/ir/185597 fatcat:i7eona4tbngb7f7yd5eywqtq7a