Developing and validating a technology-based diagnostic assessment using the evidence-centered game design approach: an example of intrapreneurship competence

Sandra Bley
2017 Empirical Research in Vocational Education and Training  
Educational experts commonly agree that tailor-made guidance is the most efficient way to foster the learning and developmental process of learners. Diagnostic assessments using cognitive diagnostic models (CDMs) have the potential to provide individual profiles of learners' strengths and weaknesses on a fine-grained level that can enable educators to assess the current position of learners. However, to obtain this necessary information a strong connection has to be made between cognition (the
more » ... ntended competence), observation (the observed learners' responses while solving the tasks), and interpretation (the inferences made based on the observed responses of learners' underlying competencies). To secure this stringent evidencebased reasoning, a principled framework for designing a technology-based diagnostic assessment is required-such as the evidence-centred game design (ECgD). Aim: With regard to a diagnostic assessment, three aspects are of particular importance according to the ECgD approach: (I) the selection of a measurable set of competence facets (so-called skills) and their grain-size, (II) the constructed pool of skill-based tasks, and (III) the clear and valid specified task to skill assignments expressed within the so-called Q matrix. The Q matrix represents the a priori assumption for running the statistical CDM-procedure for identifying learners' individual competence/skill profiles. These three prerequisites are not simply set by researchers' definition nor by experts' common sense. Rather, they require their own separate empirical studies. Hence, the focus of this paper is to evaluate the appropriateness and coherence of these three aspects (I: skill, II: tasks, and III: Q matrix). This study is a spin-off project based on the results of the governmental ASCOT research initiative on visualizing apprentices' workrelated competencies for a large-scale assessment-in particular, the intrapreneurship competence of industrial clerks. With the development of a CDM I go beyond the IRT-scaling offering the prerequisites for identifying individuals' skill profiles as a point of departure for an informative individual feedback and guidance to enhance students' learning processes. Methods: Therefore, I shall use a triangulated approach to generate three empirically based Q matrix models from different sources (experts and target-group respondents), inquiry methods (expert ratings and think-aloud studies), and methods of analyses (frequency counts and a solver-non-solver comparison). Consequently, the four single Open Access © The Author(s) 2017. This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made.
doi:10.1186/s40461-017-0049-0 fatcat:pezwswgwhnanhe55wgggevgg2a