Article

Estimating the Standard Error of the Judging in a modified-Angoff Standards Setting Procedure

---

Abstract

For a modified Angoff standards setting procedure, two methods of calculating the standard error of the judging were compared. The Central Limit Theorem (CLT) method is easy to calculate and uses readily available data. It estimates the variance of mean cut scores as a function of the variance of cut scores within a judging group, based on the independent judgements at Stage 1 of the process. Its theoretical drawback is that it is unable to take account of the effects of collaboration among the judges at Stages 2 and 3. The second method, an application of equipercentile (EQP) equating, relies on the selection of very large stable candidatures and the standardisation of the raw score distributions to remove effects associated with test difficulty. The standard error estimates were then empirically obtained from the mean cut score variation observed over a five year period. For practical purposes, the two methods gave reasonable agreement, with the CLT method working well for the top band, the band that attracts most public attention. For some bands in English and Mathematics, the CLT standard error was smaller than the EQP estimate, suggesting the CLT method be used with caution as an approximate guide only. Accessed 31,793 times on https://pareonline.net from March 01, 2004 to December 31, 2019. For downloads from January 1, 2020 forward, please click on the PlumX Metrics link to the right.

Keywords: Standards Setting, Modified Angoff Procedures, Standard Error, Central Limit Theorem, Equipercentile Equating

How to Cite:

MacCann, R. G. & Gordon, S., (2004) “Estimating the Standard Error of the Judging in a modified-Angoff Standards Setting Procedure”, Practical Assessment, Research, and Evaluation 9(1): 5. doi: https://doi.org/10.7275/n78q-6g60