Usefulness, localizability, humanness, and language-benefit: additional evaluation criteria for natural language dialogue systems

Bayan AbuShawar, Eric Atwell
<span title="2016-01-04">2016</span> <i title="Springer Nature"> <a target="_blank" rel="noopener" href="https://fatcat.wiki/container/xeix3cke4vgplj7qtbgmiixzka" style="color: black;">International Journal of Speech Technology</a> </i> &nbsp;
Human-computer dialogue systems interact with human users using natural language. We used the ALICE/AIML chatbot architecture as a platform to develop a range of chatbots covering different languages, genres, text-types, and user-groups, to illustrate qualitative aspects of natural language dialogue system evaluation. We present some of the different evaluation techniques used in natural language dialogue systems, including black box and glass box, comparative, quantitative, and qualitative
more &raquo; ... uation. Four aspects of NLP dialogue system evaluation are often overlooked: ÒusefulnessÓ in terms of a userÕs qualitative needs, Òlocalizability" to new genres and languages, "humanness" compared to humanhuman dialogues, and "language benefit" compared to alternative interfaces. We illustrated these aspects with respect to our work on machine-learnt chatbot dialogue systems; we believe these aspects are worthwhile in impressing potential new users and customers.
<span class="external-identifiers"> <a target="_blank" rel="external noopener noreferrer" href="https://doi.org/10.1007/s10772-015-9330-4">doi:10.1007/s10772-015-9330-4</a> <a target="_blank" rel="external noopener" href="https://fatcat.wiki/release/o3gpxewrl5arfnmkctsjlv5b5u">fatcat:o3gpxewrl5arfnmkctsjlv5b5u</a> </span>
<a target="_blank" rel="noopener" href="https://web.archive.org/web/20180722095947/http://eprints.whiterose.ac.uk/100801/1/Abushawar_Atwell_DialogSys_V3.pdf" title="fulltext PDF download" data-goatcounter-click="serp-fulltext" data-goatcounter-title="serp-fulltext"> <button class="ui simple right pointing dropdown compact black labeled icon button serp-button"> <i class="icon ia-icon"></i> Web Archive [PDF] <div class="menu fulltext-thumbnail"> <img src="https://blobs.fatcat.wiki/thumbnail/pdf/79/22/79227c9b4ee2b87fddfd78ac773c95e796c579bb.180px.jpg" alt="fulltext thumbnail" loading="lazy"> </div> </button> </a> <a target="_blank" rel="external noopener noreferrer" href="https://doi.org/10.1007/s10772-015-9330-4"> <button class="ui left aligned compact blue labeled icon button serp-button"> <i class="external alternate icon"></i> springer.com </button> </a>