Response type selection for chat-like spoken dialog systems based on LSTM and multi-task learning

Kengo Ohta, Ryota Nishimura, Norihide Kitaoka
2021 Speech Communication  
Please cite this article as: K. Ohta, R. Nishimura and N. Kitaoka, Response type selection for chat-like spoken dialog systems based on LSTM and multi-task learning. Speech Communication (2021), doi: https://doi. Abstract We propose a method of automatically selecting appropriate responses in conversational spoken dialog systems by explicitly determining the correct response type that is needed first, based on a comparison of the user's input utterance with many other utterances. Response
more » ... nces are then generated based on this response type designation (back channel, changing the topic, expanding the topic, etc.). This allows the generation of more appropriate responses than conventional end-to-end approaches, which only use the user's input to directly generate response utterances. As a response type selector, we propose an LSTM-based encoder-decoder framework utilizing acoustic and linguistic features extracted from input utterances. In order to extract these features more accurately, we utilize not only input utterances but also response utterances in the training corpus. To do so, multi-task J o u r n a l P r e -p r o o f Journal Pre-proof learning using multiple decoders is also investigated. To evaluate our proposed method, we conducted experiments using a corpus of dialogs between elderly people and an interviewer. Our proposed method outperformed conventional methods using either a point-wise classifier based on Support Vector Machines, or a single-task learning LSTM. The best performance was achieved when our two response type selectors (one trained using acoustic features, and the other trained using linguistic features) were combined, and multi-task learning was also performed.
doi:10.1016/j.specom.2021.07.003 fatcat:owbm3rkqqfbifcfhyclbq2azzi