Contributed Talk Sessions | Poster Sessions | All Posters | Search Papers

Poster Session C: Friday, August 15, 2:00 – 5:00 pm, de Brug & E‑Hall

Differentiating image representations in terms of their local geometry

David Lipshutz¹, Jenelle Feather², Sarah E Harvey², Alex H Williams³, Eero P Simoncelli³; ¹Baylor College of Medicine, ²Flatiron Institute, ³New York University

Presenter: David Lipshutz

Similarity between neural representations is often quantified by measuring alignment of the representations over a set of natural stimuli that are relatively far apart in stimulus space. However, systems with similar global structure can have strikingly different sensitivities to local stimulus distortions, suggesting a need for metrics that compare local sensitivities of representations. We propose a framework for comparing a set of image representations in terms of their sensitivities to local distortions. We quantify the local geometry of a representation using the Fisher information matrix, a standard statistical tool for characterizing the sensitivity to local stimulus perturbations, and use this to define a metric on the local geometry of representations near a base image. This metric may then be used to differentiate a set of representations, by finding a pair of ``principal distortions'' that maximize the variance of the representations under the metric. We apply our method to models of the early visual system and to a set of deep neural network (DNN) models.

Topic Area: Visual Processing & Computational Vision

Extended Abstract: Full Text PDF