The MIREX grand challenge: A framework of holistic user-experience evaluation in music information retrieval
Journal of the Association for Information Science and Technology
Music Information Retrieval (MIR) evaluation has traditionally focused upon system-centered approaches where components of MIR systems are evaluated against predefined datasets and golden answers (i.e., ground truth). There are two major limitations of such system-centered evaluation approaches: 1) the evaluation focuses on subtasks in music information retrieval but not entire systems; and 2) users and their interactions with MIR systems are largely excluded. This paper describes the first
... ementation of a holistic user experience evaluation in MIR, the MIREX Grand Challenge, where complete MIR systems are evaluated with user experience being the single overarching goal. It is the first time complete MIR systems have been evaluated with end-users in a realistic scenario. We present the design of the evaluation task, the evaluation criteria and a novel evaluation interface and data collection platform. This is followed by an analysis of the results, reflection of the experience and lessons learned, and plans for future directions.