Contributed Talk Sessions | Poster Sessions | All Posters | Search Papers

Poster Session B: Wednesday, August 13, 1:00 – 4:00 pm, de Brug & E‑Hall

Discovering visual categorical selectivity across the whole brain in silico using transformer-based encoders and large-scale generative models

Ethan Hwang¹, Hossein Adeli¹, Wenxuan Guo¹, Andrew Luo², Nikolaus Kriegeskorte¹; ¹Columbia University, ²University of Hong Kong

Presenter: Ethan Hwang

The human visual cortex contains several regions that selectively respond to particular categories (e.g. faces, places, bodies). However, it is unclear whether there are regions responsive to additional (possibly more complex) categories, either inside or outside the visual cortex. Jointly discovering the categories and the corresponding selective regions, without relying on the researchers' biased imagination, remains a methodological challenge. Here, we take an in-silico approach to discovering category-selective regions. We trained a state-of-the-art transformer-based encoding model that predicts neural responses from natural scenes. We then used this model to generate hypotheses about category-selectivity of different regions throughout the human brain by performing in-silico mapping, using large amounts of computation. We use diffusion-based generative models and retrieval from large image datasets to find images that maximally activate different parcels. We found many parcels with complex selectivity, transcending simple categorical concepts: scenes with multiple objects (sport events), specific subcategories (places with vanishing points or parallel lines), and specific interactions (tool use). Our study demonstrates a data-driven paradigm for discovery of visual selectivity for each region with sets of optimal images. The category-selectivity hypotheses generated can be tested in future fMRI experiments.

Topic Area: Object Recognition & Visual Attention

Extended Abstract: Full Text PDF