Contributed Talk Sessions | Poster Sessions | All Posters | Search Papers

Poster Session C: Friday, August 15, 2:00 – 5:00 pm, de Brug & E‑Hall

Constructing Representational Similarity Metrics Through Linear Decoding

Sarah E Harvey¹, David Lipshutz², Alex H Williams³; ¹Flatiron Institute, ²Baylor College of Medicine, ³New York University

Presenter: Sarah E Harvey

Neural responses encode information that is useful for a variety of downstream tasks; however, many methods for comparing neural representations do not explicitly leverage this perspective and instead highlight geometric invariances. Here, we show that many representational similarity measures can be equivalently motivated from a decoding perspective. Specifically, measures like CKA and CCA are shown to quantify the average alignment between optimal linear readouts across a distribution of decoding tasks. This approach suggests a metric on neural representations in which the distance between representations directly quantifies differences in the decoding of neural data. We demonstrate this in an ensemble of DNNs trained for image classification and human fMRI representations from the Natural Scenes Dataset. Our work demonstrates a tight link between the geometry of neural representations and the ability to linearly decode information. This perspective suggests new ways of measuring similarity between neural systems and also provides novel, unifying interpretations of existing measures.

Topic Area: Methods & Computational Tools

Extended Abstract: Full Text PDF