A Multimodal Deep Log-Based User Experience (UX) Platform for UX Evaluation

Jamil Hussain, Wajahat Ali Khan, Taeho Hur, Hafiz Syed Muhammad Bilal, Jaehun Bang, Anees Ul Hassan, Muhammad Afzal, Sungyoung Lee
2018 Sensors  
The user experience (UX) is an emerging field in user research and design, and the development of UX evaluation methods presents a challenge for both researchers and practitioners. Different UX evaluation methods have been developed to extract accurate UX data. Among UX evaluation methods, the mixed-method approach of triangulation has gained importance. It provides more accurate and precise information about the user while interacting with the product. However, this approach requires skilled
more » ... researchers and developers to integrate multiple devices, synchronize them, analyze the data, and ultimately produce an informed decision. In this paper, a method and system for measuring the overall UX over time using a triangulation method are proposed. The proposed platform incorporates observational and physiological measurements in addition to traditional ones. The platform reduces the subjective bias and validates the user's perceptions, which are measured by different sensors through objectification of the subjective nature of the user in the UX assessment. The platform additionally offers plug-and-play support for different devices and powerful analytics for obtaining insight on the UX in terms of multiple participants. The subjective aspect of the UX, however, can make UX assessment difficult. Traditional methods of UX assessment rely on self-reported measurements, usability studies (performance), and observations [6, 12] , which may be unable to uncover the true user emotional experience [3] . A common method of expressing emotional and cognitive aspects is via retrospective self-reported verbal or written questionnaires [13] [14] [15] , whereby the user is asked questions relating to their experience. However, this conventional method is highly subjective in nature and thus dependent on user interpretation, recollection, and bias [3] . Even when the questionnaire items are clear, most participants have difficulty engaging in honest and accurate introspection [3]; hence, they do not faithfully articulate their true emotions, abilities, and experiences. Meanwhile, open-ended interview methods [12] may avoid the confusion engendered by the specific-questioning process and can thus enhance the quality of user responses. Nevertheless, this method cannot completely solve the issues relating to self-reporting and self-discourse [16] . Observational methods can resolve the latter two problems, primarily when the user is unaware of the observation [17, 18] . The user observation can reveal information relating to the task/user performance, efficiency, and errors while the user interacts with the system. However, both self-reporting and observation methods are unable to determine the psychological states of the user while employing the system. Both of these methods require skilled researchers for data recording, analysis, and interpretation, the latter of which is another source of subjectivity. In short, determining how the user feels while employing the product, system, or service presents a significant hurdle for the UX evaluation. In reality, the user may have difficulty identifying, interpreting, and reporting their feelings and thoughts during or after use of the product, system, or service. Hence, in addition to self-reporting and observational measurements, UX researchers currently use physiological measurements to assess the user experience based on quantitative metrics. In general, biometric sensors can detect emotional arousal and stress, motivation, and visual attention, states that have a direct relationship with user cognitive and affective conditions [3] . For instance, an eye tracker can detect visual attention [19] [20] [21] , electroencephalography (EEG) can detect user motivations and emotional responses [22] [23] [24] , the galvanic skin response (GSR) can measure stress and arousal through skin conductivity [25, 26] , and electrocardiogram (ECG) and electromyogram (EMG) can measure stress levels and muscle-arousing activities [27, 28] . However, existing research has focused on limited methods and techniques to uncover the true experience of a user employing a product. To address the limitations of the above individualistic methods and approaches, we propose an innovative "lean UX platform", which employs a mix-method approach by combining observational, self-reported, and physiological measurements. It can evaluate the overall user experience over time by acquiring and synchronizing multimodal data while the user interacts with a product, system, or service. The rest of this paper is structured as follows: in Section 2, UX evaluation methods of related work are described. In Section 3, the proposed lean UX platform architecture is generally described. In Section 4, the overall proposed platform is presented with respect to its architecture and implementation. In Section 5, execution scenarios are presented as case studies of "mind-mining" evaluations. Section 6 presents the evaluation and discussion, and Section 7 concludes the work. Related Work Many approaches have been proposed to acquire the user experience in various ways, including the questionnaire, facial analysis, vocal analysis, biometrics, and others. We classify these user experience evaluation methods (UXEMs) into three categories: (i) self-reported measurement, whereby the participant reports their feelings and thoughts in the form of a questionnaire, survey, or poll without expert intervention; (ii) observational measurement, a non-intrusive means of observing the user while interacting with the product, system, or service; and (iii) physiological measurement, whereby sensors are mounted on the user's body for collecting physical information as quantifiable data. The following subsections detail the above categories. Sensors 2018, 18, 1622 3 of 31 Self-Reported Measurement The self-reported approach has been used for a long time as a UXEM. Different tools have been developed to gather the self-reported data from users who express their feelings about the given product, system, or service [12] . No comprehensive solutions exist for extracting the holistic UX, and every method has its positive and negative aspects [5]. For emotion measurement via self-reporting in response to a stimulus, numerous methods have been used, such as the two-dimensional (2D) emotion space (ES) [13, 29] , to gather data by moving a mouse in the 2D space in response to valence and arousal. However, it cannot be applied to low-fidelity prototypes. Similarly, expressing experiences and emotions ("3E") [14] uses a semi-structured method by providing a predefined template in which the user experience and sentiment data are entered as a daily diary. In addition, the day reconstruction method [9] is a well-known approach for capturing the user's daily experience through their reporting of three important experiences or encounters each day. However, these methods are laborious and require researchers to analyze the gathered data [3, 5, 7, 12] . Furthermore, the affect grid [15] provides a simple and easy scale for measuring affects in a 2D form, while the differential emotions scale (DES) [30] provides diverse categories of emotion to evaluate the user emotions. In addition, the Geneva emotion wheel [31] provides a wheel-shaped emotion scale through which a participant expresses their emotions, and PrEmo [32] uses cartoon animation to obtain the user's emotional responses in the form of dynamic facial, body, and vocal expressions. However, the scale is subjective. The EMO2 [33] tool provides a rating scale in one and two dimensions for emotion measurement while using the product. Emocards and Emofaces [34] use a non-verbal, quick, and easy method that employs emotion cards (cartoon faces) indicating the user emotions while using the product. However, these approaches are intrusive during the given task. Different questionnaires have been referenced in the literature for measuring various UX aspects, such as affect, aesthetics, attractiveness, pragmatics, hedonics, mental efforts, and satisfaction levels [6, 27, [35] [36] [37] [38] . Lavie and Tractinsky [39] developed an aesthetics scale for website perceived aesthetics in terms of classic and expressive aesthetics. AttrackDiff [35] and User Experience Questionnaire (UEQ) [6] facilitate a rapid assessment of the user experience by obtaining the user's expressed feelings, impressions, and attitudes after using the respective product. However, these assessments only indirectly reflect the experience, and do not focus on the actual experience. The mental effort scale [40] is an easy means of assessing how much effort is needed to complete a task; nevertheless, it requires other tools to obtain the holistic perspective. Observational Measurement
doi:10.3390/s18051622 pmid:29783712 pmcid:PMC5982399 fatcat:wltrwcuyfzebzjes5yhuo5msnm