Knowledge Grounded Multimodal Dialog Generation in Task-oriented Settings

Deeksha Varshney, Asif Ekbal Anushkha Singh
2021 Pacific Asia Conference on Language, Information and Computation  
Knowledge-grounded dialogue generation is the process of formulating an informed response based on both the conversation context and external knowledge. Multi-modality in dialogue systems has paved the way for more robust conversational bots. Any multimodal system seeks to bridge the gap between language and vision by combining information from image, audio, video, and text, all of which are often complementary. In every task-oriented dialog system, product or service attributes are crucial for
more » ... satisfying the user's needs. We propose the task of knowledge grounded response generation in a multimodal task-oriented dialog setting. We employ a multi-modal knowledge grounded incremental transformer network to generate responses that combine information from context, visuals, and external knowledge. We prepare a Knowledge Grounded Multi-Modal Dialog (KGMMD) dataset to help users in more efficient decision making. Dialog utterances, accompanying images, and knowledge in the form of hotel reviews from the hotel domain are all included in the dataset. The newly formed KGMMD dataset has been subjected to both quantitative and qualitative analysis. Evaluation results on the KGMMD datasets show that the proposed methodology outperforms the baseline models for knowledge grounded multimodal response generation. User: Hello how are you doing ? System: Hello dear. It is going well, tell me how should I assist you today? User: I'm searching for a single room in alleppey. please? System:these are the possible ones I can find in my database: 1) Kadalamma Beach Homestay, 2) Funky Art Beach House, 3) Lake Garden Resorts, 4) Ramada Alleppey, 5) Aham yoga retreat User: Ok. Give some images of the last two. System: these are some images of the different orientations possible User: I like the first one User: What is the name of the hotel with not many views but very clean and close to the station? (a) Kb: Hotel with not many views but very clean and close to the station. Staff in the hotel are just amazing, very helpful and welcoming at all times. Mariam is a start and I want to thank her for taking Care of me and my son during our stay. Managers also very professional at all times. System: Hotel with not many views but very clean and close to the station. User: What was the price of a room that wasn't refunded for a night? Kb: Man bad deal here. Room is really really small. Booked on booking.com. for 3 nights. knew right away did not want to stay. .only refunded one night. Buyer beware. Really really bad. Ready to leave without spending even one night. Rip off. This is not three stars. System: only refunded one night. User: It's great.I love it Awesome! wow amazing by the fact that it has got Pets are not allowed. facilities in this Hotel room. System: Glad to know it was helpful. It was nothing.
dblp:conf/paclic/VarshneyS21 fatcat:cnrpyknugzbthnpc64x7tgjx34