Conversational AI from an Information Retrieval Perspective: Remaining Challenges and a Case for User Simulation

Krisztian Balog
2021 Biennial Conference on Design of Experimental Search & Information Retrieval Systems  
Conversational AI is an emerging field of computer science that engages multiple research communities, from information retrieval to natural language processing to dialogue systems. Within this vast space, we focus on conversational information access, a problem that is uniquely suited to be addressed by the information retrieval community. We argue that despite the significant research activity in this area, progress is mostly limited to component-level improvements. There remains a disconnect
more » ... between current efforts and truly conversational information access systems. Apart from the inherently challenging nature of the problem, the lack of progress, in large part, can be attributed to the shortage of appropriate evaluation methodology and resources. This paper highlights challenges that render both offline and online evaluation methodologies unsuitable for this problem, and discusses the use of user simulation as a viable solution.
dblp:conf/desires/Balog21 fatcat:5iwrv7mgt5asjeo2i4rsockg6u