The Internet Archive has a preservation copy of this work in our general collections.
The file type is
In this paper, we describe a set of metrics for the evaluation of different dialogue management strategies in an implemented real-time spoken language system. ... The evaluation makes use of established metrics: the transaction success, the contextual appropriateness of system answers, the calculation of normal and correction turns in a dialogue. ... Metrics and Methods of Evaluation Implicit Recovery In evaluating a dialogue strategy for a spoken language system, attention should be paid to capture its capacity to deal with situations in which errors ...arXiv:cmp-lg/9612003v1 fatcat:cjaibzm7nbg2nif5ik3g23way4
To date, in dialogue systems research, this general methodology is not typically applied to the dialogue manager and spoken language generator. ... There is a strong relationship between evaluation and methods for automatically training language processing systems, where generally the same resource and metrics are used both to train system components ... An algorithm for automatically training a system requires a metric which the training algorithm attempts to optimize; this metric is called an objective function. ...doi:10.1007/s10579-005-2696-1 fatcat:7rhidbfxarhx5a7g3wakcrrv3y
In this paper, I discuss issues pertinent to the design of a task-based evaluation methodology for a spoken machine translation (MT) system processing human to human communication rather than human to ... I claim that system mediated human to human communication requires new evaluation criteria and metrics based on goal complexity and the speaker's prioritization of goals. ... 7 Acknowledgements I would like to thank my advisor Lori Levin, Alon Lavie, Monika Woszczyna, and Aleksandra Slavkovic for their help and suggestions with ...doi:10.3115/1034678.1034682 dblp:conf/acl/Thomas99 fatcat:syet45m5zrh5rgc3qi3lqvnkrq
Cluster-based user simulations for learning dialogue strategies and the SUPER evaluation metric. ... Quantitative evaluation of user simulation techniques for spoken dialogue systems. In Proceedings of the 6th SIGdial Workshop on Discourse and Dialogue, pages 45-54, Lisbon. ...
Evaluating the quality of a dialogue system is an understudied problem. ... Then, each class is covered with main features and the related evaluation metrics. The existence of benchmarks, suitable for the evaluation of dialogue techniques are also discussed in detail. ... We introduce the representative and prevalent benchmarks here, as listed in Table 1 . PARADISE  is the first general evaluation framework for spoken dialogue systems. ...arXiv:2108.01369v1 fatcat:ur5wmuy2rfdgnkntlh273z5l3e
This paper describes a method for creating an evaluation measure for discourse understanding in spoken dialogue systems. ... Using the multiple linear regression analysis, we have previously shown that the weighted sum of various metrics concerning dialogue states can be used for the evaluation of discourse understanding in ... Hiroshi Murase and all members of the Dialogue Understanding Research Group for helpful comments. We also thank Jun Suzuki for advise on the support vector regression. References ...doi:10.1145/1035112.1035113 dblp:journals/tslp/HigashinakaMNA04 fatcat:ferw6pqkevgp5horwak7rjwelm
Although good for exploring and experimenting with natural spoken language interactions, this approach makes the system more fragile in the presence of less-than-optimal conditions... ... Data and dialogue systems In our community it is often said: "There's no data like more data." ... "Bayesian update of dialogue state: A POMDP framework for spoken dialogue systems." Computer Speech and Language, To appear. ...doi:10.1109/asru.2009.5372951 dblp:conf/asru/Williams09 fatcat:i3cvtxkgvrgjtawbcvn3dhmil4
The dialogue strategies were learnt in a simulated environment and tested in a laboratory setting with 32 users. ... We describe an evaluation of spoken dialogue strategies designed using hierarchical reinforcement learning agents. ... In addition, this evaluation metric considers user responses with silences or incomplete dialogue acts as incoherent, the explanation for this consideration is because whatever the user said (e.g. mumbles ...doi:10.1016/j.csl.2009.07.001 fatcat:hrjxs2dv3ja4jioj4p56mz6h6q
Metrics for evaluating dialogue strategies in a spoken language system. In Proceedings of the 1995 AAAI Spring ... A proposal for incremental dialogue evaluation. In Proceedings of the DARPA Speech and Natural Language Workshop, pages 319-322. Biermann, A. W. and Philip M. Long. 1996. ...
IEEE Workshop on Automatic Speech Recognition and Understanding, 2005.
This paper presents a probabilistic method to simulate task-oriented human-computer dialogues at the intention level, that may be used to improve or to evaluate the performance of spoken dialogue systems ... In addition, we propose a dialogue similarity measure to evaluate the realism of the simulated dialogues. ... Some potential uses of the expanded corpus may be to learn optimal dialogue strategies and to evaluate spoken dialogue systems in early stages of development. ...doi:10.1109/asru.2005.1566485 fatcat:4hxmph4nrjav7ksbpzhootvqx4
In spoken dialogue system, Markov Decision Processes (MDPs) provide a formal framework for making dialogue management decisions for planning. ... Application of reinforcement learning methods in the development of dialogue strategies that support robust and efficient human-computer interaction using spoken language is a growing research area. ... EVALUATION To evaluate the effectiveness of a learnt strategy with real users, an end-to-end dialogue system with complete speech and language processing modules becomes inevitable. ...doi:10.5120/9310-3541 fatcat:ube5fvqiufbkdp7lmx2hi7nmbq
Meanwhile, several adaptation methods are proposed to adapt the pre-trained language models for multi-turn dialogue systems, in order to keep the intrinsic property of dialogue systems. ... This track incorporates new elements that are vital for the creation of a deployed task-oriented dialogue system. ... vital for the creation of a deployed task-oriented dialogue system. ...arXiv:2004.01940v1 fatcat:gaivmt6mojdbdancul2q6gk3de
Lecture Notes in Computer Science
Moreover it presents a real-life paradigm of a hands-on approach for applying usability methodologies in a spoken dialogue application environment to compare against a DTMF approach. ... This work examines the potential of usability evaluation in terms of issues and methodologies for spoken dialogue interfaces along with the appropriate designer-needs analysis. ... Acknowledgements The work described in this paper has been funded by the KAPODISTRIAS Programme of the Special Account for Research Grants, University of Athens. ...doi:10.1007/978-3-642-10308-7_36 fatcat:u6mxulxd7bhfvhg2wt5djn6exi
In this paper, we briefly summarize the role of the dialogue manager in a spoken dialogue system, give a short introduction to reinforcement-learning of dialogue management strategies and review the literature ... Dialogue Management in Spoken Dialogue Systems The field of spoken dialogue systems has seen rapid developments over the past decade, both in the academic (Jurafsky et al.). ... Henderson for many helpful discussions and suggestions. ...doi:10.1017/s0269888906000944 fatcat:oap5q7u5dvbgtdzhnrfr7vkd5a
In this paper a Bayesian Networks, BNs, approach to dialogue modelling  is evaluated in terms of a battery of both subjective and objective metrics. ... A significant effort in improving the contextual information handling capabilities of the system has been done. ... PROTOTYPE DESCRIPTION A conversational interface that allows users to drive a commercial Hifi audio system using natural language sentences is under evaluation. ...doi:10.1109/slt.2008.4777859 dblp:conf/slt/Fernandez-MartinezBFBGL08 fatcat:qrztxwuwnzhr7mph4oq6vu2lri
« Previous Showing results 1 — 15 out of 6,491 results