Automated Writing Analysis for Writing Pedagogy: From Healthy Tension to Tangible Prospects

Elena Cotos
2015 Writing & Pedagogy  
This article aims to engage specialists in writing pedagogy, assessment, genre study, and educational technologies in a constructive dialog and joint exploration of automated writing analysis as a potent instantiation of computer-enhanced assessment for learning. It recounts the values of writing pedagogy and, from this perspective, examines legitimate concerns with automated writing analysis. Emphasis is placed on the need to substantiate the construct-driven debate with systematic empirical
more » ... idence that would corroborate or refute interpretations, uses, and consequences of automated scoring and feedback tools intended for specific contexts. Such evidence can be obtained by adopting a validity argument framework. To demonstrate an application of this framework, the article presents a novel genre-based approach to automated analysis configured to support research writing and provides examples of validity evidence for using it with novice scholarly writers. Disciplines Curriculum and Instruction | Educational Assessment, Evaluation, and Research | Educational Methods | Higher Education Comments This is a manuscript of an article from Writing and Pedagogy 6 (2015). Posted with permission. Abstract This article aims to engage specialists in writing pedagogy, assessment, genre study, and educational technologies in a constructive dialog and joint exploration of automated writing analysis as a potent instantiation of computer-enhanced assessment for learning. It recounts the values of writing pedagogy and, from this perspective, examines legitimate concerns with automated writing analysis. Emphasis is placed on the need to substantiate the construct-driven debate with systematic empirical evidence that would corroborate or refute interpretations, uses, and consequences of automated scoring and feedback tools intended for specific contexts. Such evidence can be obtained by adopting a validity argument framework. To demonstrate an application of this framework, the article presents a novel genre-based approach to automated analysis configured to support research writing and provides examples of validity evidence for using it with novice scholarly writers. "Like any other technology, writing assessment and the techniques and artifacts associated with it fall prey to missing the forest for the trees when it comes time to evaluate how such technologies are put into use." (Elliot Knowles) Deane, Elliot, Ramineni, Deess, & Rudniy, 2012: 105), this technology seems to have created a widening gap rather than a link between pedagogy and assessment. This article discusses the nature of the problem, first examining the pedagogical values and the kernels of discontent with AWA, which has been discredited for its inability to read for meaning and for measuring the writing construct in a very restricted way. The intent here is not to enter the debate created by the lack of alignment between construct representation views and controversial computational realizations of aspects of writing. Rather, I would like to acknowledge legitimate concerns and to emphasize the need to substantiate the construct-driven discussion with appropriate empirical evidence of how AWA may reinforce learning and assessment. If framed as a validity argument (Kane, 2006) , such evidence can be gathered consistently to corroborate or refute interpretations, uses, and consequences intended for a given context. Most importantly, adopting a systematic validity framework, especially when automated analysis is intended to complement teaching, would allow stakeholders to engage in collaborative efforts for AWA evaluation and improved design to better support learning to write. As an example, I will describe how automated analysis can be configured for a specific instructional purpose, demonstrating a direction for future research and development through a genre-based approach.
doi:10.1558/wap.v7i2-3.26381 fatcat:6ag77uou5feszlyewyjc35h5iy