Article

Systematic Comparison of Two Approaches for Validating and Using Rater-Mediated Performance Assessments

---

Abstract

Rater-mediated performance assessments (RMPAs) are assessments that involve the use of a third-party rater to provide feedback on an individual's performance. RMPAs have been increasingly used across various contexts because of their capability to provide more accurate and reliable data than traditional self-report measures. However, many challenges still exist in effectively validating and using RMPAs, including difficulties in accounting for rater bias and other measurement errors, as well as addressing concerns around equity and fairness, especially for historically marginalized populations. This paper addresses these challenges by first discussing the methodological and/or statistical limitations of the current widely used RMPAs validation techniques based on the classical test theory (CTT) framework, including factor analysis, Cronbach’s alpha, and interrater reliability analysis, etc. Alternatively, an RMPA validation approach using Many-Facet Rasch Modeling (MFRM) is thus introduced and discussed. Methodological advantages and disadvantages of the CTT and MFRM approaches are then systematically compared from both theoretical and empirical perspectives. Finally, the commonly used techniques of a MFRM-based analysis are introduced for typical RMPA validation studies. This paper not only aims to enhance the methodological rigor of RMPAs but also seeks to contribute to the ongoing dialogue on creating more equitable and fair performance assessment practices.

Keywords: rater-mediated performance assessments, classical test theory, Many-Facet Rasch Modeling, methodological comparison

How to Cite:

Niu, C., Bradley, K. D., , R. J. & Love, A. S., (2025) “Systematic Comparison of Two Approaches for Validating and Using Rater-Mediated Performance Assessments”, Practical Assessment, Research, and Evaluation 30(1): 13. doi: https://doi.org/10.7275/pare.2042