ERQA-Plus: A Diagnostic Benchmark for Reasoning in Embodied AI

📄 arXiv: 2606.17639v1 📥 PDF

作者: Hong Yang, Basura Fernando

分类: cs.RO, cs.CV

发布日期: 2026-06-16

备注: under review at NeurIPS

🔗 代码/项目: GITHUB | HUGGINGFACE

💡 一句话要点

提出ERQA-Plus以解决现有嵌入式AI推理基准不足的问题

🎯 匹配领域: 支柱九：具身大模型 (Embodied Foundation Models)

关键词: 嵌入式AI 推理基准 视觉问答 多模态评估 机器人技术

📋 核心要点

  1. 现有的视觉和嵌入式问答基准对推理依赖的控制有限，难以有效评估嵌入式智能体的推理能力。
  2. ERQA-Plus通过结构化分类法生成问题，提供了一个多维度的推理评估框架，增强了对嵌入式推理的理解。
  3. 在基准测试中，最强模型Qwen3-VL-32B的整体准确率为83.4%，但在空间推理和程序推理等方面仍存在显著弱点。

📝 摘要（中文）

通用嵌入式智能体不仅需要物体识别，还需推理空间关系、行动、程序、人类意图、环境约束及常识后果。然而，现有的视觉和嵌入式问答基准对推理依赖的控制有限，难以区分基于视觉或语言模式匹配的快捷方式与扎根的嵌入式推理。为此，本文提出ERQA-Plus，一个用于嵌入式AI推理的诊断基准。该基准包含1766个问题-答案实例，基于711张机器人中心图像，按照感知、行动、社会互动、导航环境和上下文常识推理的结构化分类法组织。数据集通过多阶段生成和验证流程构建，结合了分类法指导的问题生成、自动质量评估、迭代修订和人工评估，以提高视觉基础、答案有效性和推理质量。

🔬 方法详解

问题定义：本文旨在解决现有嵌入式AI推理基准在推理依赖控制方面的不足，导致无法有效区分扎根推理与模式匹配的快捷方式。

核心思路：ERQA-Plus通过结构化分类法生成问题，确保问题覆盖多种推理类型，从而提供更全面的评估框架。

技术框架：数据集构建采用多阶段流程，包括分类法指导的问题生成、自动质量评估、迭代修订和人工评估，确保数据质量和推理能力的准确评估。

关键创新：ERQA-Plus的创新在于其结构化的分类法和多阶段生成流程，使得推理能力的评估更加细致和全面，超越了传统基准的局限。

关键设计：在数据集构建中，采用了自动质量判断和人工评估相结合的方式，确保问题的有效性和推理的质量，同时关注视觉基础的准确性。

🖼️ 关键图片

fig_0
fig_1
fig_2

📊 实验亮点

在基准测试中，Qwen3-VL-32B模型的整体准确率达到83.4%，但在空间推理、程序推理和意图推断等方面的表现仍显不足，显示出ERQA-Plus在评估嵌入式推理能力方面的有效性和必要性。

🎯 应用场景

ERQA-Plus可广泛应用于嵌入式AI系统的开发与评估，尤其是在机器人、自动驾驶和智能助手等领域。通过提供细致的推理能力评估，研究人员和开发者能够更好地理解和改进智能体的决策能力，从而推动智能体在复杂环境中的应用。未来，该基准有望成为嵌入式AI领域的重要参考标准。

📄 摘要（原文）

Generalist embodied agents require more than object recognition: they must reason about spatial relations, actions, procedures, human intentions, environmental constraints, and commonsense consequences from situated visual observations. Yet existing visual and embodied question answering benchmarks often provide limited control over the reasoning dependencies being tested, making it difficult to distinguish grounded embodied reasoning from shortcut-driven visual or linguistic pattern matching. We present ERQA-Plus, a diagnostic benchmark for reasoning in embodied AI. ERQA-Plus contains 1,766 question-answer instances grounded in 711 robot-centric images and organized according to a structured taxonomy spanning perceptual, action-centric, social-interaction, navigation-environmental, and contextual commonsense reasoning. The dataset is constructed using a multi-stage generation and validation pipeline that combines taxonomy-guided question generation, automatic quality judging, iterative revision, and human assessment to improve visual grounding, answer validity, and reasoning quality. We benchmark representative general-purpose vision-language models and embodied models, including LLaVA-NeXT-8B, Prismatic-7B, MiniCPM-V-4.5-8B, Qwen3-VL, RoboRefer-8B, and RoboBrain2.5-8B. Although the strongest model, Qwen3-VL-32B, achieves 83.4% overall accuracy and 61.4 SBERT score, category-level results reveal persistent weaknesses in spatial reasoning, procedural reasoning, event prediction, and intention inference. ERQA-Plus therefore provides a fine-grained evaluation framework for measuring not only whether embodied agents answer correctly, but also which forms of embodied reasoning they can and cannot perform reliably. The dataset is available https://huggingface.co/datasets/huggingdas/erqa-plus and the project page at https://github.com/LUNAProject22/erqa-plus.