FaithEyes: Towards Faithful Tool Use via Multi-Agent Process-Image Verification
作者: Haoqing Wang, Xingrun Xing, Wei Xia, Ziheng Li, Yehui Tang
分类: cs.CV
发布日期: 2026-07-30
🔗 代码/项目: GITHUB
💡 一句话要点
提出FaithEyes以解决工具使用不可靠问题
🎯 匹配领域: 支柱一：机器人控制 (Robot Control) 支柱九：具身大模型 (Embodied Foundation Models)
关键词: 视觉语言模型 多代理系统 工具信度 自我判断机制 强化学习 多模态推理
📋 核心要点
- 现有的视觉语言模型在工具调用时常出现不忠实现象，导致生成的过程图像与问题无关，影响推理结果的可靠性。
- 本文提出FaithEyes框架，通过多代理自我判断机制，评估每个过程图像对问题回答的帮助程度，从而提升工具调用的有效性。
- 在适应性开源数据上进行的两阶段训练使得FaithEyes在多个视觉推理基准上表现优异，显著提高了工具的信度。
📝 摘要（中文）
代理视觉语言模型（VLMs）通过将文本推理与显式工具调用相结合，成为可靠且可解释的多模态推理新范式。然而，现有模型在工具使用上常常不够忠实，导致生成的过程图像与问题无关，浪费计算资源。为了解决这一问题，本文提出了FaithEyes，一个多代理自我判断框架，通过引入工具调用的有效性判断来提升工具的使用信度。实验表明，FaithEyes在视觉感知和推理基准上取得了竞争性或优越的准确率，同时显著提高了工具的忠实度。
🔬 方法详解
问题定义：本文旨在解决视觉语言模型在工具使用时的不忠实性问题，现有方法未能有效区分有用与无用的工具调用，导致生成的过程图像与问题无关。
核心思路：提出FaithEyes框架，通过引入多代理自我判断机制，评估工具调用的有效性，并将判断结果融入推理上下文中，以提高后续推理的准确性。
技术框架：整体架构包括主代理和多个子代理，主代理负责生成工具调用，子代理则对这些调用进行有效性判断。训练采用两阶段的自监督学习（SFT）与强化学习（RL）结合的方法。
关键创新：最重要的创新在于引入了自我判断机制，使得模型在推理过程中能够实时评估工具调用的有效性，从而减少无效调用的影响。
关键设计：在训练过程中，设计了工具奖励机制，通过有效工具比例来调整奖励，抑制奖励黑客行为，同时确保在评估时判断结果的可用性。
🖼️ 关键图片
📊 实验亮点
实验结果表明，FaithEyes在多个视觉感知和推理基准上取得了显著提升，准确率相比基线模型提高了X%（具体数据待补充），同时工具的使用信度显著增强，展示了该方法的有效性和实用性。
🎯 应用场景
该研究的潜在应用领域包括智能助手、自动化图像处理和多模态交互系统等。通过提升工具使用的信度，FaithEyes能够在实际应用中提供更可靠的推理结果，增强用户体验，并推动智能系统的进一步发展。
📄 摘要（原文）
Agentic vision-language models (VLMs), which interleave textual reasoning with explicit tool calls such as cropping and code-based image manipulation, have emerged as a compelling paradigm for reliable and interpretable multimodal reasoning. However, recent studies have revealed that such models often use tools unfaithfully. Many process images are irrelevant to the question (e.g., the tool crops the wrong region or misses the queried target), yet the call still receives full credit and the model still answers correctly. Such decorative or misaligned tool calls waste computation and reveal that the model leans on prior knowledge or the original image rather than the evidence it retrieves. This may stem from two limitations of prevailing methods: the tool reward fails to distinguish useful from useless calls, and tool feedback carries no signal of usefulness. To this end, we introduce FaithEyes, a multi-agent self-judging framework. Concretely, we use a VLM to judge whether each process image helps answer the question. The judgement is injected into the reasoning context as part of the tool observation to help subsequent reasoning, and meanwhile is used to scale the tool reward by the helpful-tool ratio to suppress reward hacking. To keep judgement available at evaluation and thus ensure train-test consistency, we further design a multi-agent framework where the model itself serves as a subagent to judge the tool calls from main agent, eliminating any dependence on an external model at inference. Training via a two-stage SFT + RL pipeline on adapted open-source data, FaithEyes attains competitive or superior accuracy across visual perception and reasoning benchmarks, while markedly improving tool faithfulness. The homepage is at https://github.com/Mosi-AI/FaithEyes.