Contributed Talk Sessions | Poster Sessions | All Posters | Search Papers

Poster Session B: Wednesday, August 13, 1:00 – 4:00 pm, de Brug & E‑Hall

Varying sensitivity to vision and language across the STS during naturalistic social perception

Hannah Small¹, Haemy Lee Masson², Ericka Wodka, Stewart Mostofsky¹, Leyla Isik¹; ¹Johns Hopkins University, ²Durham University

Presenter: Hannah Small

Both vision and language signals contribute to real-world social processing, yet they have been mostly studied separately. To understand how these inputs are simultaneously processed, we model individual participant’s brain responses (n=34) to a naturalistic movie using vision and language deep neural network embeddings. We find that these embeddings share very little similarity in a naturalistic movie, and they both predict brain responses in the superior temporal sulcus (STS). Within STS, we identified social interaction perception and language selective regions in individual participants to examine how they process vision and language signals in the movie. We found that 1) social perception regions are best explained by vision embeddings, but are also sensitive to sentence-, but not word-, level information, 2) language regions are well predicted by speech-, word-, and sentence-level embeddings and, surprisingly, equally well predicted by vision features as language features. However, 3) language regions are exclusively sensitive to high-level visual information, whereas social perception regions (and lower level visual regions) are sensitive to both low- and high-level visual information. This work suggests that social perception and language regions both integrate visual and language signals, but the specific nature of these integrated representations vary across the STS.

Topic Area: Language & Communication

Extended Abstract: Full Text PDF