Matched Outcomes, Divergent Gaze: How Foveated MLLMs Search Compared to Humans

📄 arXiv: 2608.16514v1 📥 PDF

作者: Mohamed Amine Kerkouri, Marouane Tliba, Aladine Chetouani, Ulas Bagci, Alessandro Bruno

分类: cs.CV, cs.AI, cs.CL, cs.HC, cs.MM

发布日期: 2026-08-17

备注: Paper accepted at 3rd HCV workshop at ECCV 2026. 12 pages main text, 16 pages supp

💡 一句话要点

比较多模态大语言模型与人类视觉搜索的差异

🎯 匹配领域: 支柱九：具身大模型 (Embodied Foundation Models)

关键词: 多模态大语言模型 视觉搜索 眼动轨迹 人机交互 目标检测 效率评估

📋 核心要点

  1. 现有多模态大语言模型在模拟人类视觉搜索时存在显著差异，尤其是在注视过程的时间展开上。
  2. 本文通过对比三种MLLMs与人类的眼动轨迹，提出了一种新的评估方法，重点关注目标决策和获取效率。
  3. 实验结果表明，模型在目标检测和获取效率上超过人类，但在注视过程的自一致性上表现出低熵和大幅度的扫描路径。

📝 摘要（中文）

人类的视觉搜索是串行的，视网膜必须聚焦于候选目标以确认其存在。本文比较了三种多模态大语言模型（MLLMs）与人类在目标导向搜索中的眼动轨迹，评估了目标存在的决策、到达目标的效率以及注视过程。结果显示，模型在目标决策和获取效率上与人类相当或更优，但其注视过程并不符合人类特征。尽管匹配的视网膜输入重现了人类的注视位置，但未能模拟注视的时间展开过程，表明现有的评估指标无法有效衡量人类视觉的过程特征。

🔬 方法详解

问题定义：本文旨在探讨多模态大语言模型（MLLMs）在目标导向视觉搜索中与人类的差异，尤其是注视过程的时间特征。现有方法未能有效模拟人类的视觉搜索过程，导致评估指标的局限性。

核心思路：通过将MLLMs与人类的眼动轨迹进行比较，论文提出了一种新的评估框架，重点分析目标存在的决策、获取效率及注视过程的特征，以揭示模型与人类的根本差异。

技术框架：研究采用了COCO-Search18数据集，逐步驱动每个模型的注视，评估其在目标检测、获取效率和注视过程上的表现。主要模块包括数据输入、模型推理和结果评估。

关键创新：论文的主要创新在于揭示了MLLMs在目标决策和获取效率上超越人类的同时，其注视过程却表现出低熵和自一致性，表明其架构与人类的串行视觉搜索机制存在本质区别。

关键设计：在实验中，采用了与人类匹配的视网膜输入，确保模型在相同条件下进行评估。关键参数包括模型的注视驱动机制和评估指标的设计，确保能够准确反映目标检测和获取的效率。

🖼️ 关键图片

fig_0
fig_1
fig_2

📊 实验亮点

实验结果显示，三种MLLMs在目标检测的准确率接近人类，且在首次注视目标的效率上超过人类。然而，模型的注视过程表现出低熵和大幅度的扫描路径，表明其与人类的视觉搜索机制存在显著差异。

🎯 应用场景

该研究为多模态大语言模型在视觉搜索任务中的应用提供了新的视角，尤其是在人机交互、自动驾驶和智能监控等领域。通过理解模型与人类视觉的差异，可以推动更智能的视觉系统设计，提升其在复杂环境中的表现。

📄 摘要（原文）

Human visual search is serial: the fovea must land on a candidate to confirm it, and those landings form a scanpath. Whether multimodal large language models (MLLMs), given the same foveated input, search as humans do bears on their use as models of human vision and on attention-alignment scores. We compare three general-purpose MLLMs with human eye-movement scanpaths on goal-directed search (COCO-Search18), driving each model fixation by fixation through an identical, human-matched foveated view and assessing it along three axes: the decision of target presence, the efficiency of reaching the target, and the gaze process itself. The axes dissociate. On the decision and on target acquisition the models match or exceed humans, detecting present targets near ceiling and reaching them on the first saccade more often than people do. The gaze process is not human. Under the human-matched condition, all three share one signature: low-entropy, large-amplitude, self-consistent scanpaths that agree with themselves far more closely than two humans agree with each other. That is consistent with a single-pass, non-serial architecture rather than a limit of acuity. Matched retinal input reproduces where humans look but not how the looking unfolds in time, and no degradation regime recovers human-like search at human-like success. The gap sits on a process axis that answer-alignment and saliency metrics do not measure. Because they miss it, such metrics cannot certify human-like vision, and zero-shot models suit outcome and spatial questions but not temporal, process-level ones.