Paper

Evidence of Hierarchically-Complex Syntactic Structure Within BERT’s Word Representations

---

Abstract

Our research provides empirical support that LLM’s contextualized word embeddings have captured deep and hierarchical syntactic structure. In 2019, Hewitt and Manning found evidence that LLMs have captured features of structural dependency parses within their word representations; we extend this work by deploying their methodology on sentence structures that are differentiated only in a constituency-based account like Minimalism rather than a dependency-based account. Our novel work creates a dataset containing several carefully selected sentence structures whose dependency parses are identical, but whose constituency trees differ due to to the size of the complement (vP versus TP versus CP). We find differences in the probe's predicted distances that can only be explained if the embeddings have indeed captured some Minimalist structural difference between these sentence types. The impact of our work helps to realize Linzen's (2019) argument that linguists can further the study and understanding of LLMs and that the field of NLP provides novel tools for further linguistic research.

Keywords: syntax, dependency grammar, constituency grammar, Minimalism, LLMs, NLP, probing methods

How to Cite:

Kennedy, M. K., (2025) “Evidence of Hierarchically-Complex Syntactic Structure Within BERT’s Word Representations”, Society for Computation in Linguistics 8(1): 18. doi: https://doi.org/10.7275/scil.3103