Contributed Talk Sessions | Poster Sessions | All Posters | Search Papers

Poster Session C: Friday, August 15, 2:00 – 5:00 pm, de Brug & E‑Hall

CorText-AMA: brain-language fusion as a new tool for probing visually evoked brain responses

Victoria Bosch¹, Daniel Anthes², Adrien Doerig³, Sushrut Thorat², Peter König⁴, Tim C Kietzmann⁴; ¹Institute of Cognitive Science, Osnabrück University, Universität Osnabrück, ²University of Osnabrück, ³Freie Universität Berlin, ⁴Universität Osnabrück

Presenter: Victoria Bosch

Cognitive computational neuroscience embraces machine learning techniques to gain insight into how the brain represents and transforms visual information. Recent advances have allowed the field to move from classic category inventory approaches to more contextualized, semantic aspects, e.g. by mapping visual responses to natural scenes to corresponding language embeddings of scene captions. While the latter is powerful, single embedding vectors or captions may not fully capture the distributed cortical feature selectivity or complex spatial and semantic interactions in natural scenes. To go beyond passive representation analysis and develop interactive approaches to neural data interpretation, we extend large language models to combine a natural language interface with brain data. The resulting framework, CorText-AMA, provides an interactive chat interface that enables researchers to interrogate neural representations of natural scenes. This approach preserves semantic context while simultaneously allowing us to isolate and examine specific dimensions of brain representations. To make this possible, we combine a transformer-based multimodal model and functional brain alignment with a large instruction-finetuning dataset of question-answer pairs defined on natural scenes. The current model enables flexible probing of decodable information in visual cortex and outperforms control models. Future work will further investigate the usage of CorText-AMA as an interactive diagnostic readout that allows contrasting which questions can be answered based on neural responses in specific brain regions, and which cannot.

Topic Area: Visual Processing & Computational Vision

Extended Abstract: Full Text PDF