TDVR: Joint Text Disambiguation and Viewpoint Reasoning for Zero-Shot 3D Visual Grounding

📄 arXiv: 2608.03763v1 📥 PDF

作者: Qingxi Du, Junbo Wang, Yuke Li, Yining Zhu

分类: cs.CV

发布日期: 2026-08-04

备注: 10 pages, 5 figures, 7 tables

💡 一句话要点

提出TDVR框架以解决零样本3D视觉定位中的文本歧义与视角不足问题

🎯 匹配领域: 支柱七：动作重定向 (Motion Retargeting) 支柱九：具身大模型 (Embodied Foundation Models)

关键词: 零样本学习 3D视觉定位 文本消歧 视角推理 多模态融合

📋 核心要点

  1. 现有的零样本3D视觉定位方法在处理模糊查询文本和缺乏视角时效果显著下降，限制了其应用。
  2. TDVR框架通过无训练的推理方式，结合语义3D场景图和大型语言模型，消歧查询文本并推断最佳视角。
  3. 在ScanRefer数据集上的实验结果显示，TDVR在准确率上显著优于现有方法，提升幅度达到15.25%和14.46%。

📝 摘要（中文）

零样本3D视觉定位旨在基于文本描述和3D视觉输入定位特定对象。然而，现有方法受到模糊查询文本和视角不足的显著影响。为了解决这些问题，本文提出了TDVR，一个无训练的推理框架，能够消歧输入文本并推断准确的视角。首先，从3D点云中构建语义3D场景图。然后，将原始查询、外观和空间关系描述输入大型语言模型进行融合，从而消歧初始输入。通过链式推理生成消歧查询的结构化表示。接着，利用场景图和结构化查询作为输入，通过视角推理获得最佳视角，解决定位过程中的视角缺失问题。最后，通过计算特征向量的相似性，匹配类别文本和外观图像，最终识别目标对象。实验结果表明，TDVR在ScanRefer数据集上在Acc@0.25和Acc@0.5上分别超越现有最先进方法15.25%和14.46%。

🔬 方法详解

问题定义：本文旨在解决零样本3D视觉定位中由于模糊查询文本和视角不足导致的定位精度问题。现有方法在处理这些挑战时表现不佳，限制了其实际应用。

核心思路：TDVR框架通过构建语义3D场景图和利用大型语言模型进行文本消歧，结合视角推理来优化定位过程，从而提高模型的准确性和鲁棒性。

技术框架：整体流程包括三个主要阶段：首先从3D点云中构建语义场景图；其次将查询文本与场景信息融合以消歧；最后通过视角推理获得最佳视角并进行目标识别。

关键创新：TDVR的核心创新在于无训练的推理框架，结合了链式推理和视角推理，能够有效处理模糊文本和缺失视角的问题，这在现有方法中尚未实现。

关键设计：在设计中，采用了特征向量相似性计算来匹配类别文本和外观图像，并通过视角分数、混淆分数、类别分数和外观分数综合识别目标对象。

🖼️ 关键图片

fig_0
fig_1
fig_2

📊 实验亮点

TDVR在ScanRefer数据集上的实验结果显示，其在Acc@0.25和Acc@0.5上的准确率分别提升了15.25%和14.46%，显著超越了现有最先进的方法，证明了其在处理模糊查询和视角不足方面的有效性。

🎯 应用场景

该研究在机器人视觉、自动驾驶、增强现实等领域具有广泛的应用潜力。通过提高3D视觉定位的准确性，TDVR能够帮助系统更好地理解和互动复杂环境，推动智能设备的智能化进程。

📄 摘要（原文）

Zero-shot 3D visual grounding aims to localize specific objects based on textual descriptions and 3D visual input. However, the effectiveness of existing methods is significantly hindered by the ambiguous query text and deficient viewpoints. To address these issues, we propose TDVR, a training-free reasoning framework that disambiguates the input text and infers accurate viewpoints for zero-shot 3D visual grounding. First, we construct semantic 3D scene graph from the detected instances in the 3D point cloud. Subsequently, we put the original query, appearance and spatial relationship descriptions into the LLM for fusion, thereby disambiguating the initial input. We leverage chain-of-thought reasoning to generate the structured representation of disambiguated query. Then taking the scene graph and structured query as input, we get the optimal view via viewpoint reasoning to solve the problem of missing viewpoints during grounding. Based on the obtained optimal viewpoint, we further discriminate the distracting objects, enabling the model with the ability to distinguish similar instances. After that, we match the category text and appearance images with the query by computing the similarity of feature vectors. Finally, the target object was identified by integrating the viewpoint score, confusion score, category score, and appearance score. Compared with previous methods, our TDVR has stronger capabilities in viewpoint reasoning, similar object discrimination, and ambiguous query understanding. Experimental results on the public ScanRefer dataset show that our method outperforms the existing state-of-the-art methods by 15.25% and 14.46% in Acc@0.25 and Acc@0.5 respectively, demonstrating the effectiveness of our TDVR in addressing ambiguous query text and deficient viewpoints.