Paper

Can Syntactic Log-Odds Ratio Predict Acceptability and Satiation?

---

Abstract

The syntactic log-odds ratio (SLOR), a surprisal-based measure estimated from pretrained language models (LMs) has been proposed as a linking function for human sentence
acceptability judgments, a widespread measure of linguistic knowledge in experimental linguistics. We test this proposal in three steps: by examining whether SLOR values estimated by
GPT-2 Small predict human acceptability judgments; by asking whether satiation effects observed in human judgments are also exhibited by fine-tuned LMs; and by testing whether satiation effects generalize in qualitatively similar ways in the model compared to humans. We show that SLOR in general predicts acceptability, but there is a significant amount of variance in acceptability data that SLOR fails to capture. SLOR also fails to capture certain patterns of satiation and generalization. Our results challenge the idea that surprisal alone, via a SLOR linking function, constitutes an accurate model for human acceptability judgments.

Keywords: sentence acceptability, satiation, syntactic log-odds ratio, GPT-2

How to Cite:

Lu, J., Merchan, J., Wang, L. & Degen, J., (2024) “Can Syntactic Log-Odds Ratio Predict Acceptability and Satiation?”, Society for Computation in Linguistics 7(1), 10–19. doi: https://doi.org/10.7275/scil.2125