Contributed Talk Sessions | Poster Sessions | All Posters | Search Papers

Poster Session A: Tuesday, August 12, 1:30 – 4:30 pm, de Brug & E‑Hall

Surprising narrative events elicit convergent responses in the brain, subjective reports, and large language models

Ziwei Zhang¹, Jadyn Park, Isabella Summe, Kruthi Gollapudi, Yuan Chang Leong¹, Monica Rosenberg; ¹University of Chicago

Presenter: Ziwei Zhang

Linguistic surprise occurs when incoming linguistic information violates expectations formed from prior context. For example, when we hear a story, we are surprised when unfolding events do not align with our expectations. Here we ask whether large language models (LLMs) represent event-level surprise similarly to humans. To measure LLM surprise in two stories, we asked an LLM to generate text predictions as increasing amounts of context were revealed. For each story event, we operationalized LLM surprise as the dissimilarity between LLM’s internal embeddings of the predicted and actual text. We measured human surprise during the same events with self-reported ratings and predictions of a brain-based model of surprise applied to fMRI data. LLM surprise was significantly correlated with self-reported and brain-predicted surprise across events. This suggests that LLMs and humans predict the same events as surprising. Our findings highlight LLMs’ potential in modeling human surprise to narrative events.

Topic Area: Predictive Processing & Cognitive Control

Extended Abstract: Full Text PDF