Contributed Talk Sessions | Poster Sessions | All Posters | Search Papers

Poster Session A: Tuesday, August 12, 1:30 – 4:30 pm, de Brug & E‑Hall

Brittle Brain Encoding: Poor Out-of-Distribution Generalization Shows the Human Brain is neither a Nintendo Entertainment System nor a Four-Layer Convolutional Neural Network

Yann Harel¹, François Paugam², Marie St-Laurent¹, Pierre Bellec²; ¹Centre de Recherche de l'IUGM, ²Université de Montréal

Presenter: Yann Harel

To explore the correspondence between artificial neural networks and brain function, we tested three models—trained agent, untrained agent, and game RAM—on their capacity to use game data to predict brain activity (fMRI) in humans playing Super Mario Bros. All brain encoding models performed similarly within the training distribution (training levels), but none generalized to out-of-distribution (OOD) levels. The OOD performance drop was generally greater than the difference between models. Our results underscore how current brain encoding approaches may overstate brain-model similarity, and highlight the critical importance of evaluating generalization when using brain scores to compare models.

Topic Area: Visual Processing & Computational Vision

Extended Abstract: Full Text PDF