Proceedings

Sub-regular inductive biases in a phonological transformer

---

Abstract

Transformer neural networks can capture a wide range of linguistic structures with long-distance dependencies, but phonology is argued to be restricted to a relatively small class of sub-regular languages (Heinz 2011). We investigate the inductive biases of transformers, comparing a pretrained model (ByT5: Xue et al. 2022) with models tuned to sub-regular language classes using the Simulation-Induced Prior framework (Lindemann et al. 2024). All transformer models had a strong bias towards local processes but still learned longer-distance dependencies. SIP training may intensify this bias but is not required. Only a model trained on Tier Strictly-Local Languages, however, had a bias toward learning a plausible harmony-like representation of a long-distance effect, rather than an empirically implausible count-based rule. Moreover, we observe an asymmetry between progressive and regressive rules which is not typical of human learners. We conclude that transformers are capable of expressing inductive biases which accord with some patterns in phonological typology, but argue that assumptions of segment-by-segment sequential processing are an artifact of formal models which should not be uncritically assumed for human cognition.

Keywords: neural networks, transformers, inductive biases, long-distance processes, formal languages, sub-regular languages

How to Cite:

Elsner, M. & Black, D. S., (2026) “Sub-regular inductive biases in a phonological transformer”, Proceedings of the Annual Meetings on Phonology 2(1). doi: https://doi.org/10.7275/amphonology.3597