Targeted Syntactic Evaluation of Language Models
- Rebecca Marvin (Johns Hopkins University)
- Tal Linzen (Johns Hopkins University)
Abstract
We present a dataset for evaluating the grammatical sophistication of language models (LMs). We construct a large number of minimal pairs illustrating constraints on subject-verb agreement, reflexive anaphora and negative polarity items, in several English constructions; we expect LMs to assign a higher probability to the grammatical member of each minimal pair. An LSTM LM performed poorly in many cases. Multi-task training with a syntactic objective improved the LSTM’s accuracy, which nevertheless remained far lower than the accuracy of human participants. This suggests that there is considerable room for improvement over LSTMs in capturing syntax in an LM.
Keywords: syntax, language models, evaluation, recurrent neural networks
How to Cite:
Marvin, R. & Linzen, T., (2019) “Targeted Syntactic Evaluation of Language Models”, Society for Computation in Linguistics 2(1), 373-374. doi: https://doi.org/10.7275/p0cq-hv95
Downloads:
Download PDF