The design of methods for performance evaluation is a major open research issue in the area of spoken language dialogue systems. This paper presents the PARADISE methodology for developing predictive m o d e l s o f s p o k en dialogue performance, and shows how to evaluate the predictive p o wer and generalizability o f s u c h models. To illustrate the methodology, w e develop a number of models for predicting system usability (as measured by user satisfaction), based on the application of

more »

... ADISE to experimental data from three di erent s p o k en dialogue systems. We then measure the extent t o w h i c h the models generalize across di erent systems, di erent experimental conditions, and di erent user populations, by testing models trained on a subset of the corpus against a test set of dialogues. The results show that the models generalize well across the three systems, and are thus a rst approximation towards a general performance model of system usability.