Measuring Test-Taking Effort on Constructed-Response Items with Item Response Time and Number of Actions
Abstract
Research on methods for measuring examinee engagement with constructed-response items is limited. The present study used data from the PISA 2018 Reading domain to construct and compare indicators of test-taking effort on constructed-response items: response time, number of actions, the union (combining effortless responses detected by either response time or number of actions measures or both), and the intersection of response time and number of actions (responses identified as effortless by both response-time and number of actions measures). A 10% normative threshold identification method was used for both response time and number of actions. Pre-defined validation criteria were used to explore the validity of each of the four indicators.
Number of actions yielded a similar number of effortless responses as the union measure. Response time and intersection measures also had similar results and were related to lower disengagement than the number of actions and union indicators. With the normative threshold identification method, number of actions and the union of the two process data may result in a higher level of response misclassifications on some constructed-response items than response time and the intersection measures. Response time appears to be a more valid indicator of test-taking effort on constructed-response items than number of actions.
Keywords: test-taking effort measures, response time, number of actions, constructed-response items, PISA
How to Cite:
Ivanova, M. G. & Michaelides, M. P., (2023) “Measuring Test-Taking Effort on Constructed-Response Items with Item Response Time and Number of Actions”, Practical Assessment, Research, and Evaluation 28(1): 15. doi: https://doi.org/10.7275/pare.1921