Contributed Talk Sessions | Poster Sessions | All Posters | Search Papers

Poster Session A: Tuesday, August 12, 1:30 – 4:30 pm, de Brug & E‑Hall

Modality-Agnostic Representations are Widespread Across the Cortex

Mitja Nikolaus¹, Milad Mozafari¹, Nicholas Asher¹, Leila Reddy, Rufin VanRullen¹; ¹CNRS

Presenter: Mitja Nikolaus

Humans are able to perform tasks that require manipulation of inputs regardless of how these signals were perceived by the brain. This can be achieved thanks to representations that are agnostic to the stimulus modality. Previous work that attempted to localize such modality-agnostic representations has not yet led to conclusive results, with different studies proposing varying sets of candidate regions. These analyses have largely relied on relatively small-scale fMRI datasets with predefined sets of stimulus categories. In our work, we leveraged a new large-scale multimodal fMRI dataset of 6 subjects watching both diverse images and short text descriptions of such images to localize modality-agnostic representations. To this end, we performed a searchlight analysis with decoders trained by mapping brain activity patterns to the latent space of pretrained deep neural networks. We identified regions in which it is possible to decode both stimulus modalities in a modality-agnostic way (i.e., with a single decoder applied to brain responses from images or text). We found that large areas of the brain contain modality-agnostic representations, particularly in the left hemisphere. Our study highlights the importance of naturalistic stimuli and large-scale datasets for insightful analyses of representations in the human brain.

Topic Area: Predictive Processing & Cognitive Control

Extended Abstract: Full Text PDF