319 Hits in 5.5 sec

Interactive Evaluation of Conversational Agents: Reflections on the Impact of Search Task Design

Mateusz Dubiel, Martin Halvey, Leif Azzopardi, Sylvain Daronnat
2020 Proceedings of the 2020 ACM SIGIR on International Conference on Theory of Information Retrieval  
In the current paper we discuss findings of two Wizard of Oz studies and provide our reflections regarding the impact of different interactive search task designs on participants' performance, satisfaction  ...  In the first study, we tasked participants with finding a cheapest flight that met a certain departure time.  ...  METHOD We conducted two lab-based search studies using a Wizard of Oz (WoZ) methodology [7] , where participants interacted with a wizard (a person simulating a conversational agent) in order to find  ... 
doi:10.1145/3409256.3409814 fatcat:4f5aiwm3ufc7novuvpeqsu4lyq

ConvSearch: A Open-Domain Conversational Search Behavior Dataset [article]

Zhumin Chu, Zhihong Wang, Yiqun Liu, Yingye Huang, Min Zhang, Shaoping Ma
2022 arXiv   pre-print
To overcome these problems, we construct a Chinese Open-Domain Conversational Search Behavior Dataset (ConvSearch) based on Wizard-of-Oz paradigm in the field study scenario.  ...  We develop a novel conversational search platform to collect dialogue contents, annotate dialogue quality and candidate search results and record agent search behaviors. 25 search agents and 51 users are  ...  To overcome these problems, we construct the Chinese Open-Domain Conversational Search Behavior Dataset (ConvSearch) based on Wizard-of-Oz paradigm in the field study scenario.  ... 
arXiv:2204.02659v1 fatcat:bcwzlaliefhabfobcbdq4wu6qm

Conversational Agents for Recipe Recommendation

Sabrina Barko-Sherif, David Elsweiler, Morgan Harvey
2020 Proceedings of the 2020 Conference on Human Information Interaction and Retrieval  
By employing a Wizard of Oz (WoZ) methodology and developing a conversational framework, in this work we study how participants (n=28) interact with such a system in an attempt to solve recipe recommendation  ...  Our results are mostly encouraging for the future development of conversational agents in this context, however, they also provide insights into the complexities of building such a system that could convincingly  ...  In this work we present the results of a Wizard of Oz (WoZ) study designed to investigate how aspects of conversational interaction vary for a recommendation task in diferent conditions within the speciic  ... 
doi:10.1145/3343413.3377967 dblp:conf/chiir/Barko-SherifEH20 fatcat:koeuje3whbhdlm5xsmpxlfpiay

Wizard of Wikipedia: Knowledge-Powered Conversational agents [article]

Emily Dinan, Stephen Roller, Kurt Shuster, Angela Fan, Michael Auli, Jason Weston
2019 arXiv   pre-print
Use of knowledge has so far proved difficult, in part because of the lack of a supervised learning benchmark task which exhibits knowledgeable open dialogue with clear grounding.  ...  In open-domain dialogue intelligent agents should exhibit the use of knowledge, however there are few convincing demonstrations of this to date.  ...  the procedure in Wizard of Oz experiments (Bernsen et al., 2012) .  ... 
arXiv:1811.01241v2 fatcat:2qllsju26jhwnlbftl4y5eo7ke

Learning Adaptive Referring Expression Generation Policies for Spoken Dialogue Systems [chapter]

Srinivasan Janarthanam, Oliver Lemon
2010 Lecture Notes in Computer Science  
We present new results from a real-user evaluation of a data-driven approach to learning user-adaptive referring expression generation (REG) policies for spoken dialogue systems.  ...  We present a reinforcement learning (RL) framework in which the system learns REG policies which can adapt to unknown users online.  ...  Acknowledgements The research leading to these results has received funding from the European Community's Seventh Framework Programme (FP7/2007-2013) under grant agreement no. 216594 (CLASSiC project  ... 
doi:10.1007/978-3-642-15573-4_4 fatcat:46re3tqombbapcj3eybsusky44

Designing an interactive open-domain question answering system

2008 Natural Language Engineering  
Our Wizard-of-Oz study and final evaluation results show how the designed architecture can effectively achieve open-domain, interactive QA.  ...  Interactive question answering (QA), where a dialogue interface enables follow-up and clarification questions, is a recent although long-advocated field of research.  ...  We verify our model via a Wizard-of-Oz study and discuss the practical implementation of such model using a chatbot interface.  ... 
doi:10.1017/s1351324908004919 fatcat:yuh6qlxs6jhpxid5jexxab724q

Dialogue Actions for Natural Language Interfaces

Arne Jönsson
1995 International Joint Conference on Artificial Intelligence  
the background system, and assists the interpretation and gets eration modules The dialogue manager was designed on the basis of an investigation of empirical material collected in Wizard of Oz experiments  ...  The empirical investigations re vealed that in dialogues with database systems users specify an object, or a set of objects and ask for domain concept information, e g the value of a property of that  ...  Acknowledgments This work results from a project on Dynamic Natural-Language Understanding supported by The Swedish Council of Research in the Humanities and Social Sciences (HSFR) and The Swedish National  ... 
dblp:conf/ijcai/Jonsson95 fatcat:4mavux7f7zhyjli72pg4tnodo4

Adaptive Generation in Dialogue Systems Using Dynamic User Modeling

Srinivasan Janarthanam, Oliver Lemon
2014 Computational Linguistics  
We approach this problem using a three-step process: collecting data using a Wizard-of-Oz method, building simulated users, and learning to model and adapt to users using Reinforcement Learning techniques  ...  We also show that adaptation to users' domain knowledge results in improving task success (99.47% for the learned policy vs. 84.7% for a hand-coded baseline) and reducing dialogue time of the conversation  ...  Acknowledgments The research leading to these results has received funding from British Council (UKIERI PhD scholarships 2007-2010), the European Community's Seventh Framework Programme (FP7/2007(FP7/  ... 
doi:10.1162/coli_a_00203 fatcat:3v3wetxiabd3zhtmta3zsdflvq

Pro-active meeting assistants: attention please!

Rutger Rienks, Anton Nijholt, Paulo Barthelmess
2007 AI & Society: The Journal of Human-Centred Systems and Machine Intelligence  
We explain how to develop such assistants with respect to requirement definitions and elaborate on a set of Wizard of Oz experiments, aiming to find out in which form a meeting assistant should operate  ...  This paper gives an overview of pro-active meeting assistants, what they are and when they can be useful.  ...  Any opinions, findings and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the DARPA or the Department of Interior-National  ... 
doi:10.1007/s00146-007-0135-0 fatcat:u6dknquk3nd2zj37djpjboicpy

Evaluation of Head Gaze Loosely Synchronized With Real-Time Synthetic Speech for Social Robots

Vasant Srinivasan, Cindy L. Bethel, Robin R. Murphy
2014 IEEE Transactions on Human-Machine Systems  
requiring the operator to act as a puppeteer.  ...  This research demonstrates that robots can achieve socially acceptable interactions, using loosely synchronized head gaze-speech, without understanding the semantics of the dialog.  ...  Head gaze was implemented using the Wizard-of-Oz method [47] .  ... 
doi:10.1109/thms.2014.2342035 fatcat:u3owv47mdzggvaufd6cyh7nrm4

Technical metrics used to evaluate healthcare chatbots: Scoping review (Preprint)

Alaa Abd-Alrazaq, Zeineb Safi, Mohannad Alajlani, Jim Warren, Mowafa Househ, Kerstin Denecke
2020 Journal of Medical Internet Research  
Approaches to the evaluation of health care chatbots, however, appear to be diverse and haphazard, resulting in a potential barrier to the advancement of the field.  ...  In addition, we recommend the development of a framework of technical metrics with recommendations for specific circumstances for their inclusion in chatbot studies.  ...  games, SMS text messaging, Wizard-of-Oz, and telephones.  ... 
doi:10.2196/18301 pmid:32442157 fatcat:plog7fx2znhn7dyxl65au3azea

DoQA – Accessing Domain-Specific FAQs via Conversational QA [article]

Jon Ander Campos, Arantxa Otegi, Aitor Soroa, Jan Deriu, Mark Cieliebak, Eneko Agirre
2020 arXiv   pre-print
We present DoQA, a dataset with 2,437 dialogues and 10,917 QA pairs. The dialogues are collected from three Stack Exchange sites using the Wizard of Oz method with crowdsourcing.  ...  In addition, we introduce a more realistic information retrieval(IR) scenario where the system needs to find the answer in any of the FAQ documents.  ...  For the future, we would like to exploit the abstractive answers in our dataset, explore more sophisticated systems in both scenarios and perform user studies to study how real users interact with a conversational  ... 
arXiv:2005.01328v2 fatcat:jif6qo776vbwxogmkv6vryqz6e

How people talk when teaching a robot

Elizabeth S. Kim, Dan Leyzberg, Katherine M. Tsui, Brian Scassellati
2009 Proceedings of the 4th ACM/IEEE international conference on Human robot interaction - HRI '09  
action; and feedback, spoken in response to a completed action.  ...  naively and spontaneously use intensely affective vocalizations.  ...  We used the human judgment of a hidden Wizard of Oz operator instead of automatic affect recognition.  ... 
doi:10.1145/1514095.1514102 dblp:conf/hri/KimLTS09 fatcat:4jeumh235rbk5jtbmi2l76kw3i

Common Conversational Community Prototype: Scholarly Conversational Assistant [article]

Krisztian Balog, Lucie Flekova, Matthias Hagen, Rosie Jones, Martin Potthast, Filip Radlinski, Mark Sanderson, Svitlana Vakulenko, Hamed Zamani
2020 arXiv   pre-print
This article results from discussions of a working group at Dagstuhl Seminar 19461 on Conversational Search.  ...  This Scholarly Conversational Assistant would serve as a useful tool, a means to create datasets, and a platform for running evaluation challenges by groups across the community.  ...  Rather, corpora are often created through imagined needs (often in task-oriented Wizard-of-Oz studies), are inspired by logs, or come from crawls of community fora.  ... 
arXiv:2001.06910v1 fatcat:cvpmtmkslbahjit4tfcpdgscdy

Conversational Search (Dagstuhl Seminar 19461)

Avishek Anand, Lawrence Cavedon, Hideo Joho, Mark Sanderson, Benno Stein
2020 Dagstuhl Reports  
The ideas and findings presented in this report should serve as one of the main sources for diverse research programs on Conversational Search.  ...  Systems were invited to share the latest development in the area of Conversational Search and discuss its research agenda and future directions.  ...  We also thank the staff of Schloss Dagstuhl for providing a great venue for a successful seminar. The organisers were in part supported by JSPS KAKENHI Grant Number 19H04418.  ... 
doi:10.4230/dagrep.9.11.34 dblp:journals/dagstuhl-reports/AnandCJSS19 fatcat:ctqyitsfifcbbecio5dnx2i62y
« Previous Showing results 1 — 15 out of 319 results