Contributed Talk Sessions | Poster Sessions | All Posters | Search Papers
Representational Alignment
Contributed Talk Session: Friday, August 15, 11:00 am – 12:00 pm, Room C1.04
Evaluating Representational Similarity Measures from the Lens of Functional Correspondence
Talk 1, 11:00 am – Yiqing Bo1, Ansh Soni2, Sudhanshu Srivastava1, Meenakshi Khosla1; 1University of California, San Diego, 2University of Pennsylvania, University of Pennsylvania
Presenter: Yiqing Bo
Neuroscience and artificial intelligence (AI) both grapple with the challenge of interpreting high-dimensional neural data. Comparative analysis of such data is essential to uncover shared mechanisms and differences between these complex systems. Despite the widespread use of representational comparisons and the ever-growing landscape of comparison methods, a critical question remains: which metrics are most suitable for these comparisons? Prior work often evaluates metrics by their ability to differentiate models with varying origins (e.g., different architectures), but an alternative—and arguably more informative—approach is to assess how well these metrics distinguish models with distinct behaviors. This is crucial as representational comparisons are frequently interpreted as indicators of functional similarity in NeuroAI. To investigate this, we examine the degree of alignment between various representational similarity measures and behavioral outcomes in a suite of different downstream data distributions and tasks. We compared eight commonly used metrics in the visual domain, including alignment-based, CCA-based, inner product kernel-based, and nearest-neighbor-based methods, using group statistics and a comprehensive set of behavioral metrics. We found that metrics like the Procrustes distance and linear Centered Kernel Alignment (CKA), which emphasize alignment in the overall shape or geometry of representations, excelled in differentiating trained from untrained models and aligning with behavioral measures, whereas metrics such as linear predictivity, commonly used in neuroscience, demonstrated only moderate alignment with behavior. These findings highlight that some widely used representational similarity metrics may not directly map onto functional behaviors or computational goals, underscoring the importance of selecting metrics that emphasize behaviorally meaningful comparisons in NeuroAI research.
Rethinking Representational Alignment: Linear Probing Fails to Identify the Ground-Truth Model
Talk 2, 11:10 am – Itamar Avitan1, Tal Golan1; 1Ben Gurion University of the Negev
Presenter: Itamar Avitan
Linearly transforming stimulus representations of deep neural networks yields performant models of human similarity judgments. But can the predictive accuracy of such models identify genuine representational alignment? We conducted a model recovery study to test this empirically. We aligned 20 diverse pretrained models to 4.2 million human judgments from the THINGS-odd-one-out dataset, generated synthetic data conforming to the predictions of one of the models, and tested whether this model would re-emerge as the best predictor of the simulated data, as measured by linear probing. We found that even with large datasets, linear probing can systematically fail to recover ground-truth models. Our findings call for a reconsideration of the flexibility of model-human alignment metrics and the design of model comparison studies. $\textbf{Keywords:}$ Representational Alignment ; Model Recovery; Deep Neural Networks; Similarity Judgments
Model-brain comparison using inter-animal transforms
Talk 3, 11:20 am – Imran Thobani1, Javier Sagastuy-Brena1, Aran Nayebi2, Jacob S. Prince3, Rosa Cao, Daniel LK Yamins1; 1Stanford University, 2School of Computer Science, Carnegie Mellon University, 3Harvard University
Presenter: Imran Thobani
Artificial neural network models have emerged as promising mechanistic models of brain function, but there is little consensus on the correct method for comparing activation patterns in these models to brain responses. Drawing on recent work on mechanistic models in philosophy of neuroscience, we propose that a good comparison method should mimic the Inter-Animal Transform Class (IATC) - the strictest set of functions needed to accurately map neural responses between subjects in a population for the same brain area. Using the IATC, we can map bidirectionally between model responses and brain data, assessing how well the model can masquerade as a typical subject using the same kinds of transforms needed to map across animal subjects. We attempt to empirically identify the IATC in three settings: a simulated population of neural network models, a population of mouse subjects, and a population of human subjects. In each setting, we find that the empirically identified IATC enables accurate neural predictions while also achieving high specificity (i.e. distinguishing response patterns from different areas while strongly aligning same-area responses between subjects). In some settings, we find evidence that the IATC is shaped by specific aspects of the neural mechanism, such as the non-linear activation function. Using IATC-guided transforms, we obtain new evidence, convergent with previous findings, in favor of topographical deep neural networks (TDANNs) as models of the visual system.
Using transfer learning to identify a neural network's algorithm
Talk 4, 11:30 am – John Morrison1, Nikolaus Kriegeskorte2, Benjamin Peters3; 1Barnard College, 2Columbia University, 3University of Edinburgh, University of Edinburgh
Presenter: John Morrison
Algorithms generate input-output mappings through operations on representations. In cognitive science, we use algorithms to explain cognitive processes. For example, we use tree-search algorithms to explain planning, reinforcement learning algorithms to explain exploration, and Bayesian algorithms to explain categorization. To what extent do these algorithms describe processes in the brain? The standard method is to look for parts in the brain that correspond to the parts of an algorithm. However, we haven't found many algorithms using this method. This has led some to view cognitive science algorithms as merely normative, indicating the ideal input-output mapping without describing operations in the brain. It has led others to view these algorithms are nothing more than useful fictions; useful insofar as they allow us to predict behavior, but fictional insofar as they inaccurately describe the causes of that behavior. As an alternative, we suggest identifying a neural system's algorithm by assessing how quickly it learns alternative input-output mappings, that is, its transfer learning profile. The basic idea is that, depending on which algorithm is being used, different input-output mappings will be easier to learn, allowing us to recover its original algorithm from its transfer learning profile. We use artificial neural networks to demonstrate that this proposal productively applies to multiple networks and tasks. We conclude that transfer learning is a promising approach for integrating algorithms with neural networks and thus for integrating cognitive science with systems neuroscience and machine learning.
Models trained on infant views are more predictive of infant visual cortex
Talk 5, 11:40 am – Cliona O'Doherty1, Áine Travers Dineen2, Anna Truzzi, Graham King, Enna-Louise D'Arcy, Chiara Caldinelli3, Tamrin Holloway, Eleanor Molloy, Rhodri Cusack4; 1University of Dublin, Trinity College, 2Trinity College Dublin, 3University of Cincinnati, 4Trinity College, Dublin
Presenter: Cliona O'Doherty
The perspective of a developing infant offers unique potential when training a neural network. Egocentric video from a young child can provide ample data for representation learning in vision and language models, to only some expense of model performance. It is known that pre-trained DNNs optimised for object classification are good models of the ventral visual stream in adults, but would the same be true prior to the onset of classification behaviour? Here, we explore whether models trained on infant views are more predictive of category responses in infant ventrotemporal cortex (VTC). Using awake fMRI in a large cohort of 2-month-olds, we find that - unlike adults - features from neural networks pre-trained on infant headcam data are better models of infant VVC.