Abstract

Beyond linearity and beyond English: Language model performance on Korean NPI licensing

---

Abstract

Few studies have examined how large language models (LLMs) handle negative polarity items (NPIs) in languages other than English. This study investigates NPI licensing in Korean, an under-explored language that is subject to a well-established syntactic constraint. Our findings suggest that Korean-trained models exhibit clear difficulties in detecting this requirement when compared to human baseline judgments. This aligns with a persistent gap between human and model performance on syntax-sensitive phenomena in general (Wilcox et al. 2018; Gauthier et al. 2020; Hu et al. 2020; Warstadt et al. 2020; Lee and Schuster 2022). The present work reveals that NPI licensing is not adequately captured by the surprisal-based GPT language models: while the human results show a clear grammaticality contrast, the surprisal-based GPT models demonstrate poor and inconsistent sensitivity to this distinction. ChatGPT’s sentence-level ratings align more closely with human judgments, although its performance leaves open the question of whether it has learned the underlying syntactic generalization. This study contributes to the growing body of literature on evaluating language models on linguistic dependencies and extending the evaluation to languages beyond English.

Keywords: NPI licensing, clausemate condition, large language models, Korean

How to Cite:

Lee, S., Lee, C. & Shin, G., (2026) “Beyond linearity and beyond English: Language model performance on Korean NPI licensing”, Society for Computation in Linguistics 9(1). doi: https://doi.org/10.7275/scil.3995