EXTENDING THE RULE SPACE MODEL TO A SEMANTICALLY-RICH DOMAIN: DIAGNOSTIC ASSESSMENT IN ARCHITECTURE

Irvin R. Katz, Michael E. Martinez, Kathleen Sheehan, Kikumi K. Tatsuoka
1993 ETS Research Report Series  
Reproduction in whole or in part is permitted for any purpose of the United States Government. Approved for public release; distribution unlimited. 93-30421 93 12 1.5 044 13. ABSTRACT (Maximum 200 words) This paper presents a technique for applying the Rule Space model of cognitive diagnosis (Tatsuoka, 1983) to assessment in a semantically-rich domain. Responses to 22 architecture test items, developed to assess a range of architectural knowledge, were analyzed using Rule Space. Verbal protocol
more » ... analysis guided the construction of a model of examinee performance, consisting of processess for constructing an initial representation of an item (labeled understand), forming goals and performing actions based on those goals (solve), and determining whether goals have been attempted and satisfied (check). item attributes, derived from these processes, formed the basis for diagnosis. Successful diagnostic classifications were obtained for approximately 65Z, 90Z, and 40% of examinees based, respectively, on attributes associated with the understand, solve, and check processes of theproblem-solving model. The findings support the effectiveness of Rule Space in a complex domain and suggest directions for developing new architecture items by using attributes particularly effective at distinguishing among examinees of different ability levels. 14. SUBJECT TERMS 15. NUMBER OF PAGES diagnostic assessment; problem solving; architecture; rule space; 32 item attributes; computer-based testing. Abstract This paper presents a technique for applying the Rule Space model of cognitive diagnosis (Tamuoka, 1983) to assessment in a semantically-rich domain. Responses to 22 architecture test items, developed to assess a range of architectural knowledge, were analyzed using Rule Space. Verbal protocol analyses guided the construction of a model of examinee performance, consisting of processes for constructing an initial representation of an item (labeled und ), forming goals and performing actions based on those goals ( y., and determining whether goals have been attempted and satisfied (ch . Item attributes, derived from these processes, formed the basis for diagnosis. Our technique extends Rule Space's applicability by defining attributes in terms of item ch-arcteisti and the causal relations between characteristics and the problemsolving model Data were collected from 122 architects of various ability levels (students, architecture interns, and professional architects). Rule Space successfully classified approximately 65%, 90%, and 40% of examinees based, respectively, on attributes associated with the UndUestMId, solve, and gwk processes of the problem-solving model. The findings support the effectiveness of Rule Space in a complex domain and suggest directions for developing new architecture items by using attributes particularly effective at distinguishing among examinees of different ability levels.
doi:10.1002/j.2333-8504.1993.tb01553.x fatcat:lukow7322bhrbhw3p4y62el6z4