A Hierarchical Rater Model for Constructed Responses, with a Signal Detection Rater Model

Lawrence Decarlo, Youngkoung Kim, Matthew Johnson
2011 Journal of Educational Measurement Fall   unpublished
The hierarchical rater model (HRM) recognizes the hierarchical structure of data that arises when raters score constructed response items. In this approach, raters' scores are not viewed as being direct indicators of examinee proficiency but rather as indicators of essay quality; the (latent categorical) quality of an examinee's essay in turn serves as an indicator of the examinee's proficiency, thus yielding a hierarchical structure. Here it is shown that a latent class model motivated by
more » ... l detection theory (SDT) is a natural candidate for the first level of the HRM, the rater model. The latent class SDT model provides measures of rater precision and various rater effects, above and beyond simply severity or leniency. The HRM-SDT model is applied to data from a large-scale assessment and is shown to provide a useful summary of various aspects of the raters' performance.
fatcat:f62s5do5hjb7houegfnatgch4a