Article

Statistical Analyses with Sampling Weights in Large-Scale Assessment and Survey

---

Abstract

Large-scale assessment and survey (LSAS) studies have been increasingly utilized to address important research questions. As LSAS data adopt multi-stage complex sampling designs with unequal probabilities of selection, sampling weights need to be used. However, it is unclear how to use different sampling weights variables in statistical analysis using LSAS data. Thus far, research evidence and practical guidelines have been scarce and inconsistent. Using data from Canada, Italy, and Lithuania in PISA and TIMSS that represent two main sampling frameworks in LSAS, this study examined unweighted and weighted analyses and compared weighted single-level versus multi-level models with different sampling weights variables (e.g., student weights, house weights, senate weights, replicate weights, and school weights). Findings reveal that weighted estimates were different from unweighted estimates, and statistical significance of model analyses may vary for some variables in different countries and different LSAS data. For weighted multilevel models, three approaches (i.e., size scaling, effective scaling, school weights only) generated more similar results. Practical recommendations are provided.

Keywords: complex sampling weights, statistical analysis, large-scale assessment and survey

How to Cite:

Shen, T., (2024) “Statistical Analyses with Sampling Weights in Large-Scale Assessment and Survey”, Practical Assessment, Research, and Evaluation 29(1): 4. doi: https://doi.org/10.7275/pare.2014