The MIREX grand challenge: A framework of holistic user-experience evaluation in music information retrieval

Xiao Hu, Jin Ha Lee, David Bainbridge, Kahyun Choi, Peter Organisciak, J. Stephen Downie
2015 Journal of the Association for Information Science and Technology  
Music Information Retrieval (MIR) evaluation has traditionally focused upon system-centered approaches where components of MIR systems are evaluated against predefined datasets and golden answers (i.e., ground truth). There are two major limitations of such system-centered evaluation approaches: 1) the evaluation focuses on subtasks in music information retrieval but not entire systems; and 2) users and their interactions with MIR systems are largely excluded. This paper describes the first
more » ... ementation of a holistic user experience evaluation in MIR, the MIREX Grand Challenge, where complete MIR systems are evaluated with user experience being the single overarching goal. It is the first time complete MIR systems have been evaluated with end-users in a realistic scenario. We present the design of the evaluation task, the evaluation criteria and a novel evaluation interface and data collection platform. This is followed by an analysis of the results, reflection of the experience and lessons learned, and plans for future directions.
doi:10.1002/asi.23618 fatcat:6y34cd7jircmxal2tzotgxdqam