Article

Examining the Impact of a Consensus Approach to Content Alignment Studies

---

Abstract

Although both content alignment and standard-setting procedures rely on content-expert panel judgements, only the latter employs discussion among panel members. This study employed a modified form of the Webb methodology to examine content alignment for twelve tests administered as part of the Massachusetts Comprehensive Assessment System (MCAS). This modification required panel members to discuss items for which there was no consensus regarding the item’s depth of knowledge or targeted standard. After the discussion, panel members were allowed to change their original ratings. The number of changes that occurred were analyzed considering the number of items discussed and the size of the panel. Moreover, we evaluated the impact these changes had on the overall judgments of alignment as reported by Webb’s Web Alignment Tool (WAT). Findings suggest that discussion among panel members between rating rounds positively increased agreement among panel members’ ratings but had minimal effects on the overall judgments of content alignment for 11 of the 12 tests evaluated.

Keywords: content alignment, validity, standard setting

How to Cite:

Russell, M., (2020) “Examining the Impact of a Consensus Approach to Content Alignment Studies”, Practical Assessment, Research, and Evaluation 25(1): 4. doi: https://doi.org/10.7275/5na1-gk87