Integrating Real-Time Drawing and Writing Diagnostic Models: An Evidence-Centered Design Framework for Multimodal Science Assessment [chapter]

Andy Smith, Osman Aksit, Wookhee Min, Eric Wiebe, Bradford W. Mott, James C. Lester
2016 Lecture Notes in Computer Science  
Interactively modeling science phenomena enables students to develop rich conceptual understanding of science. While this understanding is often assessed through summative, multiple-choice instruments, science notebooks have been used extensively in elementary and secondary grades as a mechanism to promote and reveal reflection through both drawing and writing. Although each modality has been studied individually, obtaining a comprehensive view of a student's conceptual understanding requires
more » ... alyses of knowledge represented across both modalities. Evidence-centered design (ECD) provides a framework for diagnostic measurement of data collected from student interactions with complex learning environments. This work utilizes ECD to analyze a corpus of elementary student writings and drawings collected with a digital science notebook. First, a competency model representing the core concepts of each exercise, as well as the curricular unit as a whole, was constructed. Then, evidence models were created to map between student written and drawn artifacts and the shared competency model. Finally, the scores obtained using the evidence models were used to train a deep-learning based model for automated writing assessment, as well as to develop an automated drawing assessment model using topological abstraction. The findings reveal that ECD provides an expressive unified framework for multimodal assessment of science learning with accurate predictions of student learning. tutoring systems for science education should support multimodal assessment of both student drawing and student writing [5] . Evidence-centered design (ECD) provides a systematic approach to designing and developing assessments [6] . ECD identifies multiple phases in the design process, each with its own explicit goals. These phases include the creation of a Competency Model, an Evidence Model, and a Task Model that operate in concert to recognize evidence of conceptual understanding from student work. For multimodal assessment, ECD can provide a systematic way of mapping between learning goals and student artifacts from various modalities that show evidence of student learning. Of particular interest is how ECD might provide a unified framework for assessing both written and drawn artifacts of student work for formative purposes. This paper introduces a new ECD-based framework for multimodal science assessment. First, we use a multimodal approach to ECD to define a competency model and a multimodal evidence model for elementary science to understand how conceptual understanding about magnetism is revealed in both drawing and writing tasks. Specifically we aim to evaluate student writings and drawings using a common competency model that contributes to a deeper understanding of the relative contributions of the two modalities. Second, with the long-term goal of integrating multimodal assessments into an ITS, we present computational models for evaluating student writings and drawings in real-time and compare their predictive accuracy to expert human scorings. The findings reveal that ECD provides a unified framework for multimodal assessment of science learning with accurate predictions of student learning.
doi:10.1007/978-3-319-39583-8_16 fatcat:mqmrfhdtsjhghdobynqows2tm4