The measurement of text quality: current methods and open challenges

There is a general agreement that writing skills are essential in various areas of life, such as academic and professional achievement, lifelong learning, participation in society, etc (Valtin et al., 2016). However, there is limited understanding of what constitutes a well-written text, how writing can be evaluated, to what extent the criteria that guide the evaluation can be articulated and standardised. This lack of clarity can place a significant burden on educators, who bear the responsibility of evaluating students’ texts and provide constructive feedback. In some cases, it may also prompt concerns about the validity of high-stakes assessments that rely on the evaluation of written texts (Kobrin et al., 2011). The present study aims to explore how writing quality is operationalised in research and how scholars ensure the validity and reliability of its measurement. It is based on the analysis of 60 studies published in two leading journals focusing on research on writing, Assessing Writing and Journal of Second Language Writing. The analyses show that the most established way to operationalise the construct of writing quality is by means of holistic or analytic evaluation scales. The reliability of the measurements is in most cases ensured by rater training and the calculation of inter-rater reliability or agreement; intra-rater reliability is rarely controlled for. The validity of the instruments is in most cases assumed rather than critically examined or discussed.