Developing Reliable Lab Rubrics Using Only Two Columns

Joshua Enszer
2019 ASEE Annual Conference & Exposition Proceedings   unpublished
Dr. Joshua Enszer is an associate professor in Chemical and Biomolecular Engineering at the University of Delaware. He has taught core and elective courses across the curriculum, from introduction to engineering science and material and energy balances to process control, capstone design, and mathematical modeling of chemical and environmental systems. His research interests include technology and learning in various incarnations: electronic portfolios as a means for assessment and professional
more » ... development, implementation of computational tools across the chemical engineering curriculum, and game-based learning. Abstract Rubrics have often been touted as an effective tool to communicate expectations and save time grading. One area of study is the improvement of rubrics to increase inter-rater reliability; that is, creating consistent rubrics such that multiple assessors are likely to assign the same score or designation to the same work. The number of columns or "standard levels" within a rubric is often up to debate: for a given criteria, should work be assessed on a three-, four-, or five-point scale, or should another strategy be adopted altogether? One drawback to increasing the number of levels in a rubric is that it may become more likely for multiple assessors to use the rubric to assign different ratings. Another task that comes with increased levels is the writing of descriptions that accurately communicate the kind of work that merits each level. Could it be effective to structure rubrics using only two levels? In this work, we will summarize some of the literature on the development of rubrics, and then describe our process of creating a "twocolumn" rubric -one that only describes excellent and minimally acceptable performances. We will share examples of how we apply these two-column rubrics in our junior-and senior-level chemical engineering laboratory courses. We explain our algorithm for using the two-column rubrics, including how faculty, teaching assistants, and students are trained to apply the algorithm. Finally, we conducted inter-rater reliability analysis for an example assignment and found modest improvement in agreement between assessors compared to previous evaluation methods. We conclude with our next steps in our development and revision of these rubrics.
doi:10.18260/1-2--32644 fatcat:o3xqqzwfuvgeznowkm3kz2woxy