Contributed Talk Sessions | Poster Sessions | All Posters | Search Papers

Poster Session C: Friday, August 15, 2:00 – 5:00 pm, de Brug & E‑Hall

Sparse Encoding of Grammatical Gender in LSTM Language Models

Priyanka Sukumaran¹, Conor Houghton¹, Nina Kazanina²; ¹University of Bristol, ²University of Geneva

Presenter: Conor Houghton

Neural network language models excel at capturing the complexities of natural language, yet their internal representations remain poorly understood. A key question is whether such models form structured, human-like abstractions that support generalization. We investigate how LSTM language models encode grammatical gender—an ideal test case, as gender is lexically fixed and generally not inferable from semantics. We focus on long-distance dependencies and various gender agreement configurations. We conduct single-unit ablation to identify neurons critical for grammatical gender agreement. Across eight LSTM models, we find between one and five units whose removal significantly disrupts performance—by over 40\% in some constructions involving gender-interfering nouns. These units are essential for both noun-adjective and noun-past-participle gender agreement. Neuron activity analyses reveal that these units exhibit category-specific effects, with some showing a preference for default gender forms, such as masculine nouns. Our findings show that LSTMs develop sparse and structured representations of grammatical gender, reminiscent of grandmother cells in neuroscience. These results suggest that abstract grammatical categories can emerge naturally in LSTM training. More broadly, this work contributes to our understanding of how language models encode linguistic structure, with implications for model interpretability and parallels between artificial and biological computation.

Topic Area: Language & Communication

Extended Abstract: Full Text PDF