Contributed Talk Sessions | Poster Sessions | All Posters | Search Papers

Poster Session B: Wednesday, August 13, 1:00 – 4:00 pm, de Brug & E‑Hall

NeuroAdapter: Visual Reconstruction with Masked Brain Representation

Hossein Adeli¹, Wenxuan Guo¹, Pinyuan Feng¹, Ethan Hwang¹, Fan L. Cheng¹, Nikolaus Kriegeskorte¹; ¹Columbia University

Presenter: Hossein Adeli

Recent advances in generative decoding models have shown that complex visual scenes can be reconstructed from brain activity. However, current models rely on an intermediate step that maps the brain data to rich image and text feature spaces, resulting in overly large and computationally intensive models. This intermediate process may also cause loss of information deriving from the selectivity and receptive field location of individual brain units. In this work, we explore the capabilities of visual decoding in the absence of intermediate representations. We propose NeuroAdaptor, a simple modular framework that directly encodes the neural data from different brain regions to condition the diffusion process. To avoid overfitting, our model incorporates a random token-masking strategy. We train our model on the 7T-fMRI Natural Scenes Dataset (NSD) and evaluate it on multiple metrics. NeuroAdapter excels at capturing high-level semantic visual content from fMRI signals, outperforming more complex models. Our model demonstrates a promising direction for scaling decoding models up to whole-brain image reconstruction.

Topic Area: Object Recognition & Visual Attention

Extended Abstract: Full Text PDF