Article

The Miscalculation of Interrater Reliability: A Case Study Involving the AAC&U VALUE Rubrics

---

Abstract

Institutional assessment of student learning objectives has become a fact-of-life in American higher education and the Association of American Colleges and Universities’ (AAC&U) VALUE Rubrics have become a widely adopted evaluation and scoring tool for student work. As faculty from a variety of disciplines, some less familiar with the psychometric literature, are drawn into assessment roles, it is important to point out two easily made but serious errors in what might appear to be one of the more straightforward assessments of measurement quality—interrater reliability. The first error which can occur when a third rater is brought in to adjudicate a discrepancy in the scores reported by an initial two raters has been well-documented in the literature but never before illustrated with AAC&U rubrics. The second error is to cease training before the raters have demonstrated a satisfactory level of interrater reliability. This research note describes an actual case study in which the interrater reliability of the AAC&U rubrics was incorrectly reported and when correctly reported found to be inadequate. The note concludes with recommendations for the correct measurement of interrater reliability. Accessed 1,575 times on https://pareonline.net from December 08, 2017 to December 31, 2019. For downloads from January 1, 2020 forward, please click on the PlumX Metrics link to the right.

Keywords: Evaluation Methods, Research Methodology, Statistical Analysis

How to Cite:

Szafran, R., (2017) “The Miscalculation of Interrater Reliability: A Case Study Involving the AAC&U VALUE Rubrics”, Practical Assessment, Research, and Evaluation 22(1): 11. doi: https://doi.org/10.7275/xfp0-8228