Filters








6,491 Hits in 5.6 sec

Metrics for Evaluating Dialogue Strategies in a Spoken Language System [article]

Morena Danieli, Elisabetta Gerbino
1996 arXiv   pre-print
In this paper, we describe a set of metrics for the evaluation of different dialogue management strategies in an implemented real-time spoken language system.  ...  The evaluation makes use of established metrics: the transaction success, the contextual appropriateness of system answers, the calculation of normal and correction turns in a dialogue.  ...  Metrics and Methods of Evaluation Implicit Recovery In evaluating a dialogue strategy for a spoken language system, attention should be paid to capture its capacity to deal with situations in which errors  ... 
arXiv:cmp-lg/9612003v1 fatcat:cjaibzm7nbg2nif5ik3g23way4

Can We Talk? Methods for Evaluation and Training of Spoken Dialogue Systems

Marilyn A. Walker
2005 Language Resources and Evaluation  
To date, in dialogue systems research, this general methodology is not typically applied to the dialogue manager and spoken language generator.  ...  There is a strong relationship between evaluation and methods for automatically training language processing systems, where generally the same resource and metrics are used both to train system components  ...  An algorithm for automatically training a system requires a metric which the training algorithm attempts to optimize; this metric is called an objective function.  ... 
doi:10.1007/s10579-005-2696-1 fatcat:7rhidbfxarhx5a7g3wakcrrv3y

Designing a task-based evaluation methodology for a spoken machine translation system

Kavita Thomas
1999 Proceedings of the 37th annual meeting of the Association for Computational Linguistics on Computational Linguistics -  
In this paper, I discuss issues pertinent to the design of a task-based evaluation methodology for a spoken machine translation (MT) system processing human to human communication rather than human to  ...  I claim that system mediated human to human communication requires new evaluation criteria and metrics based on goal complexity and the speaker's prioritization of goals.  ...  7 Acknowledgements I would like to thank my advisor Lori Levin, Alon Lavie, Monika Woszczyna, and Aleksandra Slavkovic for their help and suggestions with  ... 
doi:10.3115/1034678.1034682 dblp:conf/acl/Thomas99 fatcat:syet45m5zrh5rgc3qi3lqvnkrq

Page 510 of Computational Linguistics Vol. 34, Issue 4 [page]

2008 Computational Linguistics  
Cluster-based user simulations for learning dialogue strategies and the SUPER evaluation metric.  ...  Quantitative evaluation of user simulation techniques for spoken dialogue systems. In Proceedings of the 6th SIGdial Workshop on Discourse and Dialogue, pages 45-54, Lisbon.  ... 

How to Evaluate Your Dialogue Models: A Review of Approaches [article]

Xinmeng Li, Wansen Wu, Long Qin, Quanjun Yin
2021 arXiv   pre-print
Evaluating the quality of a dialogue system is an understudied problem.  ...  Then, each class is covered with main features and the related evaluation metrics. The existence of benchmarks, suitable for the evaluation of dialogue techniques are also discussed in detail.  ...  We introduce the representative and prevalent benchmarks here, as listed in Table 1 . PARADISE [67] is the first general evaluation framework for spoken dialogue systems.  ... 
arXiv:2108.01369v1 fatcat:ur5wmuy2rfdgnkntlh273z5l3e

Evaluating discourse understanding in spoken dialogue systems

Ryuichiro Higashinaka, Noboru Miyazaki, Mikio Nakano, Kiyoaki Aikawa
2004 ACM Transactions on Speech and Language Processing  
This paper describes a method for creating an evaluation measure for discourse understanding in spoken dialogue systems.  ...  Using the multiple linear regression analysis, we have previously shown that the weighted sum of various metrics concerning dialogue states can be used for the evaluation of discourse understanding in  ...  Hiroshi Murase and all members of the Dialogue Understanding Research Group for helpful comments. We also thank Jun Suzuki for advise on the support vector regression. References  ... 
doi:10.1145/1035112.1035113 dblp:journals/tslp/HigashinakaMNA04 fatcat:ferw6pqkevgp5horwak7rjwelm

Spoken dialogue systems: Challenges, and opportunities for research

Jason D. Williams
2009 2009 IEEE Workshop on Automatic Speech Recognition & Understanding  
Although good for exploring and experimenting with natural spoken language interactions, this approach makes the system more fragile in the presence of less-than-optimal conditions...  ...  Data and dialogue systems In our community it is often said: "There's no data like more data."  ...  "Bayesian update of dialogue state: A POMDP framework for spoken dialogue systems." Computer Speech and Language, To appear.  ... 
doi:10.1109/asru.2009.5372951 dblp:conf/asru/Williams09 fatcat:i3cvtxkgvrgjtawbcvn3dhmil4

Evaluation of a hierarchical reinforcement learning spoken dialogue system

Heriberto Cuayáhuitl, Steve Renals, Oliver Lemon, Hiroshi Shimodaira
2010 Computer Speech and Language  
The dialogue strategies were learnt in a simulated environment and tested in a laboratory setting with 32 users.  ...  We describe an evaluation of spoken dialogue strategies designed using hierarchical reinforcement learning agents.  ...  In addition, this evaluation metric considers user responses with silences or incomplete dialogue acts as incoherent, the explanation for this consideration is because whatever the user said (e.g. mumbles  ... 
doi:10.1016/j.csl.2009.07.001 fatcat:hrjxs2dv3ja4jioj4p56mz6h6q

Page 1995 of Computational Linguistics Vol. 23, Issue 1 [page]

1997 Computational Linguistics  
Metrics for evaluating dialogue strategies in a spoken language system. In Proceedings of the 1995 AAAI Spring  ...  A proposal for incremental dialogue evaluation. In Proceedings of the DARPA Speech and Natural Language Workshop, pages 319-322. Biermann, A. W. and Philip M. Long. 1996.  ... 

Human-computer dialogue simulation using hidden Markov models

H. Cuayahuitl, S. Renals, O. Lemon, H. Shimodaira
2005 IEEE Workshop on Automatic Speech Recognition and Understanding, 2005.  
This paper presents a probabilistic method to simulate task-oriented human-computer dialogues at the intention level, that may be used to improve or to evaluate the performance of spoken dialogue systems  ...  In addition, we propose a dialogue similarity measure to evaluate the realism of the simulated dialogues.  ...  Some potential uses of the expanded corpus may be to learn optimal dialogue strategies and to evaluate spoken dialogue systems in early stages of development.  ... 
doi:10.1109/asru.2005.1566485 fatcat:4hxmph4nrjav7ksbpzhootvqx4

A Learning Automata based Solution for Optimizing Dialogue Strategy in Spoken Dialogue System

G. Kumaravelan, R. Sivakumar
2012 International Journal of Computer Applications  
In spoken dialogue system, Markov Decision Processes (MDPs) provide a formal framework for making dialogue management decisions for planning.  ...  Application of reinforcement learning methods in the development of dialogue strategies that support robust and efficient human-computer interaction using spoken language is a growing research area.  ...  EVALUATION To evaluate the effectiveness of a learnt strategy with real users, an end-to-end dialogue system with complete speech and language processing modules becomes inevitable.  ... 
doi:10.5120/9310-3541 fatcat:ube5fvqiufbkdp7lmx2hi7nmbq

Pre-Trained and Attention-Based Neural Networks for Building Noetic Task-Oriented Dialogue Systems [article]

Jia-Chen Gu, Tianda Li, Quan Liu, Xiaodan Zhu, Zhen-Hua Ling, Yu-Ping Ruan
2020 arXiv   pre-print
Meanwhile, several adaptation methods are proposed to adapt the pre-trained language models for multi-turn dialogue systems, in order to keep the intrinsic property of dialogue systems.  ...  This track incorporates new elements that are vital for the creation of a deployed task-oriented dialogue system.  ...  vital for the creation of a deployed task-oriented dialogue system.  ... 
arXiv:2004.01940v1 fatcat:gaivmt6mojdbdancul2q6gk3de

Spoken Dialogue Interfaces: Integrating Usability [chapter]

Dimitris Spiliotopoulos, Pepi Stavropoulou, Georgios Kouroupetroglou
2009 Lecture Notes in Computer Science  
Moreover it presents a real-life paradigm of a hands-on approach for applying usability methodologies in a spoken dialogue application environment to compare against a DTMF approach.  ...  This work examines the potential of usability evaluation in terms of issues and methodologies for spoken dialogue interfaces along with the appropriate designer-needs analysis.  ...  Acknowledgements The work described in this paper has been funded by the KAPODISTRIAS Programme of the Special Account for Research Grants, University of Athens.  ... 
doi:10.1007/978-3-642-10308-7_36 fatcat:u6mxulxd7bhfvhg2wt5djn6exi

A survey of statistical user simulation techniques for reinforcement-learning of dialogue management strategies

JOST SCHATZMANN, KARL WEILHAMMER, MATT STUTTLE, STEVE YOUNG
2006 Knowledge engineering review (Print)  
In this paper, we briefly summarize the role of the dialogue manager in a spoken dialogue system, give a short introduction to reinforcement-learning of dialogue management strategies and review the literature  ...  Dialogue Management in Spoken Dialogue Systems The field of spoken dialogue systems has seen rapid developments over the past decade, both in the academic (Jurafsky et al.).  ...  Henderson for many helpful discussions and suggestions.  ... 
doi:10.1017/s0269888906000944 fatcat:oap5q7u5dvbgtdzhnrfr7vkd5a

Evaluation of a spoken dialogue system for controlling a Hifi audio system

F. Fernandez Martinez, J. Blazquez, J. Ferreiros, R. Barra, J. Macias-Guarasa, J.M. Lucas-Cuesta
2008 2008 IEEE Spoken Language Technology Workshop  
In this paper a Bayesian Networks, BNs, approach to dialogue modelling [1] is evaluated in terms of a battery of both subjective and objective metrics.  ...  A significant effort in improving the contextual information handling capabilities of the system has been done.  ...  PROTOTYPE DESCRIPTION A conversational interface that allows users to drive a commercial Hifi audio system using natural language sentences is under evaluation.  ... 
doi:10.1109/slt.2008.4777859 dblp:conf/slt/Fernandez-MartinezBFBGL08 fatcat:qrztxwuwnzhr7mph4oq6vu2lri
« Previous Showing results 1 — 15 out of 6,491 results