ViewMind3D: Modular View-Aware Inference for Training-Free 3D-QA

📄 arXiv: 2607.28442v1 📥 PDF

作者: Ping-Kun Chiang, Kun-Ru Wu, Po-han Li, Sandeep Chinchali, Ufuk Topcu, Yu-Chee Tseng

分类: cs.CV

发布日期: 2026-07-30

💡 一句话要点

提出ViewMind3D以解决训练依赖的3D问答问题

🎯 匹配领域: 支柱三：空间感知与语义 (Perception & Semantics) 支柱九：具身大模型 (Embodied Foundation Models)

关键词: 3D问答 无训练推理 模块化框架 空间推理 视觉语言模型

📋 核心要点

  1. 现有的3D问答方法依赖于昂贵的训练和标注，限制了其在实际场景中的应用。
  2. ViewMind3D提出了一种无训练的模块化框架，通过分解任务实现高效的3D空间推理。
  3. 实验结果显示，ViewMind3D在空间定位问题上表现优异，整体准确率达到50.8%，在ScanQA上CIDEr得分为73.4。

📝 摘要（中文）

近年来，大型语言模型和视觉语言模型的进步为3D问答（3D-QA）带来了新的可能性。然而，现有方法通常依赖于昂贵的3D特定训练或微调，限制了其可扩展性和实际应用。本文提出了ViewMind3D，一个完全无训练且模块化的框架，能够在无需完整3D重建的情况下，对场景的多视角观察进行3D空间推理。该框架将3D-QA任务分解为四个可解释的组件：问题驱动的多视角选择、基于语言的视觉定位、鸟瞰视角的空间上下文编码，以及基于角色的结构化答案生成。实验结果表明，ViewMind3D在ScanQA和SQA3D上表现出色，尤其在空间定位问题上有显著提升。

🔬 方法详解

问题定义：本文旨在解决现有3D问答方法依赖昂贵训练和标注的问题，限制了其实际应用的可行性。

核心思路：ViewMind3D通过模块化设计，分解3D-QA任务为多个可解释的组件，从而实现无需训练的高效推理。

技术框架：该框架包括四个主要模块：问题驱动的多视角选择、语言条件的视觉定位、鸟瞰视角的空间上下文编码，以及基于角色的结构化答案生成。

关键创新：最重要的创新在于其完全无训练的设计，利用通用的语言模型和视觉模型进行模块化推理，区别于传统的训练依赖方法。

关键设计：在设计中，采用了问题驱动的视角选择策略和基于语言的视觉提示，确保了推理过程的结构化和可解释性。

🖼️ 关键图片

fig_0
img_1
img_2

📊 实验亮点

实验结果表明，ViewMind3D在ScanQA和SQA3D数据集上表现优异，整体准确率达到50.8%，在空间定位问题上有显著提升，CIDEr得分为73.4，显示出其在3D问答任务中的有效性。

🎯 应用场景

ViewMind3D的研究成果在机器人感知和智能体交互中具有广泛的应用潜力，能够提升机器人在复杂环境中的空间理解能力，促进人机协作和自主导航等领域的发展。

📄 摘要（原文）

Recent advances in large language models (LLMs) and vision-language models (VLMs) have enabled new possibilities for 3D question answering (3D-QA), a key capability for embodied AI and robotic perception. However, most existing methods rely on 3D-specific training or fine-tuning with costly annotations, limiting their scalability and real-world applicability. We present \textbf{ViewMind3D}, a fully training-free and modular framework for 3D spatial reasoning over multi-view observations of a scene without requiring complete 3D reconstruction. The framework decomposes the 3D-QA task into four interpretable components: (1) question-driven multi-view selection, (2) guided visual grounding with language-conditioned object cues, (3) spatial context encoding via a bird's-eye-view (BEV) viewpoint indicator, and (4) structured answer generation through role-based reasoning. This design enables structured, robust, and interpretable reasoning without requiring model tuning. Experimental results on ScanQA and SQA3D show that ViewMind3D achieves competitive performance compared to prior training-free and fine-tuned 3D-LLMs. In particular, our method improves performance on spatially grounded question types, such as ``What'' questions in SQA3D, while maintaining strong overall accuracy (50.8\%) and achieving 73.4 CIDEr on ScanQA. These results demonstrate that effective 3D reasoning can be achieved through modular orchestration of general-purpose LLMs and VLMs for robotic perception in real-world environments.