Rapid Diagnostic Approach If instructional techniques and formats need to be dynamically tailored to levels of learner expertise (Kalyuga, 2005A), a question of considerable interest is how to measure the levels of expertise rapidly and in real time. A rapid diagnostic approach has been developed to evaluate what is the highest level of organised knowledge structures (if any) a person is capable of retrieving and applying to the briefly presented task? With the rapid verification method,

more »

... s were presented with a series of possible (correct and incorrect) intermediate steps of a task solution and asked to rapidly verify the correctness of each step. The rapid verification method was used for optimizing levels of instructional guidance in an adaptive computerbased tutor for vector addition motion problems in kinematics (Kalyuga, 2005B). To validate the rapid verification method, two experiments were designed to investigate if rapid verification tests of levels of learner expertise in two different domains would correlate highly with a more traditional evaluation of expertise based on observations of participants' problem solving steps using video-recordings and concurrent verbal reports. 33 university students from Sydney participated in the study (in Experiment 1, data for one student was lost due to a software problem). Experiment 1 The test items represented a class of tasks in kinematics (vector addition motion problems). A typical task in this domain requires adding two vectors that are positioned at a certain angle to each other, for example, A sea wave is travelling at 8 m/s towards the beach. A swimmer moves at 3 m/s in a direction perpendicular to the direction of the wave. What is the velocity of the swimmer relative to the ground? During the rapid verification test, more knowledgeable learners presumably should be better able to rapidly construct, integrate, and recognize intermediate solution moves than less knowledgeable learners. An evaluation of cognitive load using a simple subjective rating scale was also included in the procedure to provide another indicator of levels of learner expertise in addition to the test performance scores. A Pearson product-moment correlation r(31) = .71, p <.01, between scores for the traditional and rapid tests was obtained, with 95% confidence interval extending from 0.48 to 0.85, suggesting a high degree of the predictive validity for the rapid test. A Pearson product-moment correlation between average ratings of task difficulty for the traditional and rapid tests was r(31) = .67, p <.01, with 95% confidence interval extending from 0.42 to 0.83. Test time for the rapid method was reduced by a factor of 3.22 in comparison with the time for the traditional test. The average response time (in seconds) for verification tasks was M = 3.04, SD = 0.81, indicating that students actually responded rapidly, as they were asked to do in pre-test instructions. Experiment 2 This experiment was designed as a correlation study in the task domain of transforming graphs of linear and quadratic functions in mathematics. Two tasks asked students to transform a provided graph of the basic line y = x into graphs of more complex lines, for example, y = -2x + 3 and y = x -2. The following two tasks asked students to transform a provided graph of the basic line y = x² into graphs of more complex quadratic functions, for example, y = -x² and y = 2(x -2)². The tasks required application of two or three of the following operations: flipping a graph because of the minus sign in front of x or x² (the negative slope); squeezing (expanding) a graph toward (from) the yaxis according to the value of a coefficient in front of x or x² (more or less than 1); and horizontal/vertical shifting. A Pearson product-moment correlation r(32) = .75, p <.01, between scores for the traditional and rapid tests was obtained, with 95% confidence interval extending from 0.55 to 0.87, suggesting a high degree of the predictive validity for the rapid test. A Pearson product-moment correlation between average ratings of task difficulty for the traditional and rapid tests was r(32) = .82, p <.01, with 95% confidence interval extending from 0.66 to 0.91. Test time for the rapid method was reduced by a factor of 3.52 in comparison with the time for the traditional test. The results of both experiments in this study indicated significant correlations between learners' performance scores and difficulty ratings on the rapid test tasks and traditional measures of expertise, demonstrating a high degree of external validity of the rapid verification method.