Co-Designing a Real-Time Classroom Orchestration Tool to Support Teacher–AI Complementarity

Kenneth Holstein, Bruce M. McLaren, Vincent Aleven
2019 Journal of Learning Analytics  
Involving stakeholders throughout the creation of new educational technologies can help ensure their usefulness and usability in real-world contexts. However, given the complexity of learning analytics (LA) systems, it can be challenging to meaningfully involve non-technical stakeholders throughout their design and development. This article presents a detailed case study of the iterative co-design of Lumilo, a wearable, real-time learning analytics tool for teachers working in AI-enhanced K-12
more » ... lassrooms. In the process, we argue that the co-design of LA systems requires new kinds of prototyping methods. We introduce one of our own prototyping methods, REs, to address unique challenges of co-prototyping data-driven algorithmic systems such as LA tools. This work presents the first end-to-end demonstration in the literature of how non-technical stakeholders can participate throughout the whole design process for a complex LA systemfrom early generative phases to the selection and tuning of analytics to evaluation in real-world contexts. We conclude with a summary of methodological recommendations for future LA co-design efforts. Notes for Practice • The field of Learning Analytics (LA) is beginning to explore new methods and strategies for codesigning LA tools with critical stakeholders such as teachers, students, and parents. Effectively implementing LA co-design processes requires drawing upon design and prototyping methods from a range of disciplines, and in some cases creating new ones. However, demonstrations of successful co-design processes for LA tools remain very rare in the literature. • This article provides an end-to-end demonstration and methodological recommendations for how nontechnical stakeholders can meaningfully participate throughout the design of a complex LA tool. For example, when implementing co-design processes, designers of LA tools should centre initial discussions on stakeholder needs, rather than specific analytics, visualizations, or other technical considerations. In addition, to ensure the usefulness and usability of the resulting designs, designers should scaffold stakeholders in reflecting on how particular LA displays might inform (or fail to inform) instructional decision-making and action in the context of specific tasks and scenarios. • The present case study illustrates the importance of carefully considering which roles to augment, which to automate, and which to leave alone, when designing LA tools for use in social contexts such as classrooms. Working closely with key stakeholders such as teachers can help designers to better understand their values, and nuances of the contexts in which LA technologies will be used. In turn, this can help in understanding where automation or augmentation via LA can help more than hurt.
doi:10.18608/jla.2019.62.3 fatcat:yqr6tgoh4rg7fj2ya3zajfnaxu