Contributed Talk Sessions | Poster Sessions | All Posters | Search Papers

Poster Session B: Wednesday, August 13, 1:00 – 4:00 pm, de Brug & E‑Hall

Disentangling belief and strategy in natural visual search

Hyunwoo Gu¹, Justin L Gardner¹; ¹Stanford University

Presenter: Hyunwoo Gu

Our beliefs and strategies are not always aligned. For example, in visual search, an aligned strategy would be to choose to look where one believes the target to be (belief maximization). However, previous studies using simple stimuli have found that human search performance is comparable to an ideal observer which maximizes information (Najemnik & Geisler, 2009), a strategy which can sometimes select gaze to locations where little is known instead of where the target is believed to be. In naturalistic settings, however, visual search poses additional challenges; for example, targets can take on many possible appearances, and object affordances can suggest actions such as reaching, which may influence gaze strategies. While widely used predictive models of visual saliency (Itti & Koch, 2001; Droste, Jiao, & Noble, 2020; Kümmerer, Bethge, & Wallis, 2022; Ding et al., 2022; Hosseini, Kazerouni, Akhavan, Brudno, & Taati, 2024; Yang et al., 2024) have achieved impressive accuracy in predicting human fixations from image features, they do not model belief or strategy. Moreover, these models are not normative as they do not specify the ideal criteria that can be compared to human performance, prohibiting a principled way to assess optimal belief propagation and visual search strategy. To address these challenges and characterize the gaze selection strategy, we generalized an ideal observer model (Najemnik & Geisler, 2005) to natural images with an explicit modular structure of belief and strategy. Across a publicly available dataset (COCO-search18) and a dataset we collected, we found that estimated strategies do not align with the beliefs, deviating from an intuitive, maximum-seeking strategy. Furthermore, we explicitly tested whether people’s choice of eye movements matches their beliefs using a novel gaze-contingent paradigm, and we found that where people shift their gaze to and where they believe the target to be can differ substantially. Taken together, these results suggest that people tend to prioritize information-seeking over belief maximization in naturalistic visual search.

Topic Area: Object Recognition & Visual Attention

Extended Abstract: Full Text PDF