A practical approach to measuring user engagement with the refined user engagement scale (UES) and new UES short form

Heather L. O'Brien, Paul Cairns, Mark Hall
2018 International Journal of Human-Computer Studies  
User engagement (UE) and its measurement have been of increasing interest in human-computer interaction (HCI). The User Engagement Scale (UES) is one tool developed to measure UE, and has been used in a variety of digital domains. The original UES consisted of 31-items and purported to measure six dimensions of engagement: aesthetic appeal, focused attention, novelty, perceived usability, felt involvement, and endurability. A recent synthesis of the literature questioned the original
more » ... . Further, the ways in which the UES has been implemented in studies suggests there may be a need for a briefer version of the questionnaire and more effective documentation to guide its use and analysis. This research investigated and verified a four-factor structure of the UES and proposed a Short Form (SF). We employed contemporary statistical tools that were unavailable during the UES ' development to re-analyze the original data, consisting of 427 and 779 valid responses across two studies, and examined new data ( N = 344) gathered as part of a three-year digital library project. In this paper we detail our analyses, present a revised long and short form (SF) version of the UES, and offer guidance for researchers interested in adopting the UES and UES-SF in their own studies. (M. Hall). URL: http://www.heatherobrien.arts.ubc.ca (H.L. O 'Brien), https://www-users.cs.york.ac.uk/~pcairns/ (P. Cairns) • self-reports such as questionnaires, interviews, diary entries and verbal elicitation. All methodological approaches have their advantages and limitations with respect to use with specific populations, settings, and time scales, from a single user-computer interaction to longitudinal observations. In addition, measures may capture interactions formatively or summatively, and subjectively or objectively . In general, there has been advocacy for multiple measures and mixed methods to reliably and validly capture constructs such as user engagement. This requires attention to the robustness of individual measures, as well as to triangulating multiple measures. Our work is concerned with the User Engagement Scale (UES), a 31-item experiential questionnaire. The UES (or items derived from it) has been used to evaluate engagement in a range of settings: information search, online news, online video, education, and consumer applications, haptic technologies, social networking systems, and video games (see ( O 'Brien, 2016b ) for an overview of this work). Although there is evidence to suggest that the UES is a reliable and valid means of
doi:10.1016/j.ijhcs.2018.01.004 fatcat:dbkkk4d57zeolaph7vagdnjjbu