The Internet Archive has a preservation copy of this work in our general collections.
The file type is application/pdf.
Filters
Metrics for Evaluating Dialogue Strategies in a Spoken Language System
[article]
1996
arXiv
pre-print
Preserved Fulltext
In this paper, we describe a set of metrics for the evaluation of different dialogue management strategies in an implemented real-time spoken language system. ...
The evaluation makes use of established metrics: the transaction success, the contextual appropriateness of system answers, the calculation of normal and correction turns in a dialogue. ...
Metrics and Methods of Evaluation
Implicit Recovery In evaluating a dialogue strategy for a spoken language system, attention should be paid to capture its capacity to deal with situations in which errors ...
arXiv:cmp-lg/9612003v1
fatcat:cjaibzm7nbg2nif5ik3g23way4
Can We Talk? Methods for Evaluation and Training of Spoken Dialogue Systems
2005
Language Resources and Evaluation
Preserved Fulltext
A copy of this work was available on the public web and has been preserved in the Wayback Machine. The capture dates from 2017; you can also visit the original URL.
The file type is application/pdf.
To date, in dialogue systems research, this general methodology is not typically applied to the dialogue manager and spoken language generator. ...
There is a strong relationship between evaluation and methods for automatically training language processing systems, where generally the same resource and metrics are used both to train system components ...
An algorithm for automatically training a system requires a metric which the training algorithm attempts to optimize; this metric is called an objective function. ...
doi:10.1007/s10579-005-2696-1
fatcat:7rhidbfxarhx5a7g3wakcrrv3y
Designing a task-based evaluation methodology for a spoken machine translation system
1999
Proceedings of the 37th annual meeting of the Association for Computational Linguistics on Computational Linguistics -
Preserved Fulltext
A copy of this work was available on the public web and has been preserved in the Wayback Machine. The capture dates from 2017; you can also visit the original URL.
The file type is application/pdf.
In this paper, I discuss issues pertinent to the design of a task-based evaluation methodology for a spoken machine translation (MT) system processing human to human communication rather than human to ...
I claim that system mediated human to human communication requires new evaluation criteria and metrics based on goal complexity and the speaker's prioritization of goals. ...
7 Acknowledgements I would like to thank my advisor Lori Levin, Alon Lavie, Monika Woszczyna, and Aleksandra Slavkovic for their help and suggestions with ...
doi:10.3115/1034678.1034682
dblp:conf/acl/Thomas99
fatcat:syet45m5zrh5rgc3qi3lqvnkrq
Page 510 of Computational Linguistics Vol. 34, Issue 4
[page]
2008
Computational Linguistics
Preserved Fulltext
The Internet Archive has digitized a microfilm copy of this work. It may be possible to borrow a copy for reading.
Cluster-based user simulations for learning dialogue strategies and the SUPER evaluation metric. ...
Quantitative evaluation of user simulation techniques for spoken dialogue systems. In Proceedings of the 6th SIGdial Workshop on Discourse and Dialogue, pages 45-54, Lisbon. ...
How to Evaluate Your Dialogue Models: A Review of Approaches
[article]
2021
arXiv
pre-print
Preserved Fulltext
A copy of this work was available on the public web and has been preserved in the Wayback Machine. The capture dates from 2021; you can also visit the original URL.
The file type is application/pdf.
Evaluating the quality of a dialogue system is an understudied problem. ...
Then, each class is covered with main features and the related evaluation metrics. The existence of benchmarks, suitable for the evaluation of dialogue techniques are also discussed in detail. ...
We introduce the representative and prevalent benchmarks here, as listed in Table 1 . PARADISE [67] is the first general evaluation framework for spoken dialogue systems. ...
arXiv:2108.01369v1
fatcat:ur5wmuy2rfdgnkntlh273z5l3e
Evaluating discourse understanding in spoken dialogue systems
2004
ACM Transactions on Speech and Language Processing
Preserved Fulltext
A copy of this work was available on the public web and has been preserved in the Wayback Machine. The capture dates from 2005; you can also visit the original URL.
The file type is application/pdf.
This paper describes a method for creating an evaluation measure for discourse understanding in spoken dialogue systems. ...
Using the multiple linear regression analysis, we have previously shown that the weighted sum of various metrics concerning dialogue states can be used for the evaluation of discourse understanding in ...
Hiroshi Murase and all members of the Dialogue Understanding Research Group for helpful comments. We also thank Jun Suzuki for advise on the support vector regression.
References ...
doi:10.1145/1035112.1035113
dblp:journals/tslp/HigashinakaMNA04
fatcat:ferw6pqkevgp5horwak7rjwelm
Spoken dialogue systems: Challenges, and opportunities for research
2009
2009 IEEE Workshop on Automatic Speech Recognition & Understanding
Preserved Fulltext
A copy of this work was available on the public web and has been preserved in the Wayback Machine. The capture dates from 2011; you can also visit the original URL.
The file type is application/pdf.
Although good for exploring and experimenting with natural spoken language interactions, this approach makes the system more fragile in the presence of less-than-optimal conditions... ...
Data and dialogue systems In our community it is often said: "There's no data like more data." ...
"Bayesian update of dialogue state: A POMDP framework for spoken dialogue systems." Computer Speech and Language, To appear. ...
doi:10.1109/asru.2009.5372951
dblp:conf/asru/Williams09
fatcat:i3cvtxkgvrgjtawbcvn3dhmil4
Evaluation of a hierarchical reinforcement learning spoken dialogue system
2010
Computer Speech and Language
Preserved Fulltext
A copy of this work was available on the public web and has been preserved in the Wayback Machine. The capture dates from 2017; you can also visit the original URL.
The file type is application/pdf.
The dialogue strategies were learnt in a simulated environment and tested in a laboratory setting with 32 users. ...
We describe an evaluation of spoken dialogue strategies designed using hierarchical reinforcement learning agents. ...
In addition, this evaluation metric considers user responses with silences or incomplete dialogue acts as incoherent, the explanation for this consideration is because whatever the user said (e.g. mumbles ...
doi:10.1016/j.csl.2009.07.001
fatcat:hrjxs2dv3ja4jioj4p56mz6h6q
Page 1995 of Computational Linguistics Vol. 23, Issue 1
[page]
1997
Computational Linguistics
Preserved Fulltext
The Internet Archive has digitized a microfilm copy of this work. It may be possible to borrow a copy for reading.
Metrics for evaluating dialogue strategies in a spoken language system. In Proceedings of the 1995 AAAI Spring ...
A proposal for incremental dialogue evaluation. In Proceedings of the DARPA Speech and Natural Language Workshop, pages 319-322.
Biermann, A. W. and Philip M. Long. 1996. ...
Human-computer dialogue simulation using hidden Markov models
2005
IEEE Workshop on Automatic Speech Recognition and Understanding, 2005.
Preserved Fulltext
A copy of this work was available on the public web and has been preserved in the Wayback Machine. The capture dates from 2013; you can also visit the original URL.
The file type is application/pdf.
This paper presents a probabilistic method to simulate task-oriented human-computer dialogues at the intention level, that may be used to improve or to evaluate the performance of spoken dialogue systems ...
In addition, we propose a dialogue similarity measure to evaluate the realism of the simulated dialogues. ...
Some potential uses of the expanded corpus may be to learn optimal dialogue strategies and to evaluate spoken dialogue systems in early stages of development. ...
doi:10.1109/asru.2005.1566485
fatcat:4hxmph4nrjav7ksbpzhootvqx4
A Learning Automata based Solution for Optimizing Dialogue Strategy in Spoken Dialogue System
2012
International Journal of Computer Applications
Preserved Fulltext
A copy of this work was available on the public web and has been preserved in the Wayback Machine. The capture dates from 2017; you can also visit the original URL.
The file type is application/pdf.
In spoken dialogue system, Markov Decision Processes (MDPs) provide a formal framework for making dialogue management decisions for planning. ...
Application of reinforcement learning methods in the development of dialogue strategies that support robust and efficient human-computer interaction using spoken language is a growing research area. ...
EVALUATION To evaluate the effectiveness of a learnt strategy with real users, an end-to-end dialogue system with complete speech and language processing modules becomes inevitable. ...
doi:10.5120/9310-3541
fatcat:ube5fvqiufbkdp7lmx2hi7nmbq
Pre-Trained and Attention-Based Neural Networks for Building Noetic Task-Oriented Dialogue Systems
[article]
2020
arXiv
pre-print
Preserved Fulltext
A copy of this work was available on the public web and has been preserved in the Wayback Machine. The capture dates from 2020; you can also visit the original URL.
The file type is application/pdf.
Meanwhile, several adaptation methods are proposed to adapt the pre-trained language models for multi-turn dialogue systems, in order to keep the intrinsic property of dialogue systems. ...
This track incorporates new elements that are vital for the creation of a deployed task-oriented dialogue system. ...
vital for the creation of a deployed task-oriented dialogue system. ...
arXiv:2004.01940v1
fatcat:gaivmt6mojdbdancul2q6gk3de
Spoken Dialogue Interfaces: Integrating Usability
[chapter]
2009
Lecture Notes in Computer Science
Preserved Fulltext
A copy of this work was available on the public web and has been preserved in the Wayback Machine. The capture dates from 2020; you can also visit the original URL.
The file type is application/pdf.
Moreover it presents a real-life paradigm of a hands-on approach for applying usability methodologies in a spoken dialogue application environment to compare against a DTMF approach. ...
This work examines the potential of usability evaluation in terms of issues and methodologies for spoken dialogue interfaces along with the appropriate designer-needs analysis. ...
Acknowledgements The work described in this paper has been funded by the KAPODISTRIAS Programme of the Special Account for Research Grants, University of Athens. ...
doi:10.1007/978-3-642-10308-7_36
fatcat:u6mxulxd7bhfvhg2wt5djn6exi
A survey of statistical user simulation techniques for reinforcement-learning of dialogue management strategies
2006
Knowledge engineering review (Print)
Preserved Fulltext
A copy of this work was available on the public web and has been preserved in the Wayback Machine. The capture dates from 2017; you can also visit the original URL.
The file type is application/pdf.
In this paper, we briefly summarize the role of the dialogue manager in a spoken dialogue system, give a short introduction to reinforcement-learning of dialogue management strategies and review the literature ...
Dialogue Management in Spoken Dialogue Systems The field of spoken dialogue systems has seen rapid developments over the past decade, both in the academic (Jurafsky et al.). ...
Henderson for many helpful discussions and suggestions. ...
doi:10.1017/s0269888906000944
fatcat:oap5q7u5dvbgtdzhnrfr7vkd5a
Evaluation of a spoken dialogue system for controlling a Hifi audio system
2008
2008 IEEE Spoken Language Technology Workshop
Preserved Fulltext
A copy of this work was available on the public web and has been preserved in the Wayback Machine. The capture dates from 2017; you can also visit the original URL.
The file type is application/pdf.
In this paper a Bayesian Networks, BNs, approach to dialogue modelling [1] is evaluated in terms of a battery of both subjective and objective metrics. ...
A significant effort in improving the contextual information handling capabilities of the system has been done. ...
PROTOTYPE DESCRIPTION A conversational interface that allows users to drive a commercial Hifi audio system using natural language sentences is under evaluation. ...
doi:10.1109/slt.2008.4777859
dblp:conf/slt/Fernandez-MartinezBFBGL08
fatcat:qrztxwuwnzhr7mph4oq6vu2lri
« Previous
Showing results 1 — 15 out of 6,491 results