cIBS: an ideal Bayesian observer for visual search in natural scenes

BUJIA G; SCLAR M; VITA S; SOLOVEY G; KAMIENKOWSKI JE

Congreso; Society for Neuroscience; 2021

Visual search is a vital task that humans perform systematically on an everyday basis. In the last decades, thanks to the advance of deep neural networks, there was a large development of models that predict the most likely gaze fixation locations when observing a scene (saliency maps). Today, one of the biggest challenges in the field is to go beyond these saliency maps to predict the exact sequence of fixations and tasks different from simple observation. In visual search tasks, Bayesian observers have been proposed to model the visual search behavior as an active sampling process. In this process, during each fixation, humans incorporate new information and update the probability of finding a target at every location. Here, we combine these approaches for visual search in natural images and propose a model to predict the whole scanpath. Our Ideal Bayesian Searcher (IBS) uses a saliency map as prior and computes the most likely next location given all the previous fixations, considering visual properties of the target and the scene. We collected eye-movement visual search data (N=57) in 134 natural indoor scenes and compared different variants of the model and its parameters.  Firstly, different state-of-the-art saliency maps performed similarly to previous reports only on the first fixations. After the third fixation, their performance rapidly decayed to almost chance. This suggests that saliency maps are not able to capture fixations distributions when top-down information is critical. Secondly, we introduced different variants of the IBS model to deal with natural scenes, as the priors and measure of similarity between the target and the whole scene. We showed that IBS models using saliency-based priors produced scanpaths that were more similar to humans than using simpler priors, both in the percentage of target found as a function of the fixation rank and the scanpath similarity, reproducing the entire sequence of eye movements. Thirdly, the IBS model presented here behaves more similar to humans than other state-of-the-art scanpath prediction models based on deep neural networks in the present dataset with scenes of interiors with multiple distractors. Overall, our results and others showed that the Bayesian observer is a natural framework to model the top-down processes, such as the integration of information and predictions, that influence visual search and also guide eye movements beyond the saliency models.