Article

A Comparison of Subject Matter Experts’ Perceptions and Job Analysis Surveys

---

Abstract

Two common approaches for performing job analysis in credentialing programs are committee-based methods, which rely solely on subject matter experts’ judgments, and task inventory surveys. This study evaluates how well subject matter experts’ perceptions coincide with task inventory survey results for three credentialing programs. Results suggest that subject matter expert ratings differ in systematic ways from task inventory survey results and that task lists generated based solely on subject matter experts’ intuitions generally lead to narrower task lists. Results also indicated that there can be key differences for procedures and non-procedures, with subject matter experts’ judgments often tending to exhibit lower agreement levels with task inventory survey results for procedures than for non-procedures. We recommend that organizations performing job analyses think very carefully before relying solely on subject matter experts’ judgments as their primary method of job analysis. Accessed 1,252 times on https://pareonline.net from September 06, 2018 to December 31, 2019. For downloads from January 1, 2020 forward, please click on the PlumX Metrics link to the right.

Keywords: Test Construction

How to Cite:

Wyse, A. E. & Babcock, B., (2018) “A Comparison of Subject Matter Experts’ Perceptions and Job Analysis Surveys”, Practical Assessment, Research, and Evaluation 23(1): 10. doi: https://doi.org/10.7275/erf8-ca22