Article

An R Package for Optimizing the Composite Reliability in Multivariate Nested Designs

---

Abstract

Background

The reliability of assessment tools is a crucial aspect of monitoring student performance in various educational settings. It ensures that the assessment outcomes accurately reflect a student's true level of performance. However, when assessments are combined, each assessment contributes as a data point towards the overall performance of the student on the overarching educational framework. In that case, determining composite reliability can be challenging, especially for naturalistic and unbalanced datasets, as is common in programmatic assessment.

Results

This paper introduces the CompositeReliability package in R, designed to estimate composite reliability using multivariate generalizability theory and enhance the analysis of assessment data. The package produces extensive Generalizability and Decision study results with graphical interpretations. Composite reliability incorporates weights and covariance to integrate results across assessment tools. Weights can be optimized to minimize standard error of measurement or maximize reliability.

Conclusions

Overall, the package’s flexible use and optimization empower assessment tailoring and robust insights into student performance. The approach is suitable for programmatic assessment. The package facilitates reliable, comprehensive evaluation across diverse assessments.

Keywords: Multivariate generalizability theory, Composite reliability, SEM, Programmatic assessment, R

How to Cite:

Moonen - van Loon, J. M. & Donkers, J., (2025) “An R Package for Optimizing the Composite Reliability in Multivariate Nested Designs”, Practical Assessment, Research, and Evaluation 30(1): 4. doi: https://doi.org/10.7275/pare.2045