Stream3Dv2: Geometric-Semantic Fusion Enhanced Streaming Zero-Shot 3D Scene Understanding

📄 arXiv: 2608.21136v1 📥 PDF

作者: Jie Xu, Na Zhao

分类: cs.CV

发布日期: 2026-08-21

🔗 代码/项目: GITHUB

💡 一句话要点

提出Stream3Dv2以解决流媒体RGB-D输入处理和噪声分割问题

🎯 匹配领域: 支柱三：空间感知与语义 (Perception & Semantics) 支柱九：具身大模型 (Embodied Foundation Models)

关键词: 流媒体3D感知 几何-语义融合 点云优化 开放词汇理解 无训练框架

📋 核心要点

  1. 现有的开放词汇零-shot 3D场景理解方法在处理流媒体RGB-D输入时效率低下，且对噪声分割掩码的鲁棒性不足。
  2. 论文提出的Stream3Dv2框架通过嵌套局部到历史架构和几何-语义融合机制，提升了流媒体3D感知的稳定性和效率。
  3. 实验结果显示，Stream3Dv2在多个公共数据集上显著优于现有基线，证明了其在3D分割和检测任务中的有效性。

📝 摘要（中文）

近年来，基于视觉基础模型的开放词汇零-shot 3D场景理解成为数据密集型监督方法的有力替代。然而，这些模型在实际应用中受到流媒体RGB-D输入处理效率低下和对噪声2D分割掩码脆弱性的严重限制。为了解决这些关键问题，我们提出了Stream3Dv2，一个旨在实现稳健流媒体3D感知的新型无训练框架。Stream3Dv2通过独特的嵌套局部到历史架构处理顺序数据，捕捉多视图一致性，同时避免高计算开销，以支持及时响应。核心是引入全面的几何-语义融合机制，通过显式利用语义指导，解决几何噪声和语义模糊，将3D分割形式化为点集合并和划分问题。我们还提出了一种创新的基于流形距离的点云优化策略，利用局部流形图进行点到流形的优化，减轻了欧几里得距离度量导致的边界划分失败，并动态激活和更新历史实例以实现快速的流形间优化。大量实验表明，Stream3Dv2在基础开放词汇流媒体3D分割和检测中始终优于现有基线。

🔬 方法详解

问题定义：本论文旨在解决流媒体RGB-D输入处理效率低和对噪声2D分割掩码脆弱性的问题。现有方法在实时应用中表现不佳，无法满足实际需求。

核心思路：Stream3Dv2通过嵌套局部到历史架构处理顺序数据，结合几何-语义融合机制，显著提升了3D感知的鲁棒性和效率。此设计旨在捕捉多视图一致性，同时降低计算开销。

技术框架：Stream3Dv2的整体架构包括数据流处理模块、几何-语义融合模块和点云优化模块。数据流处理模块负责接收和处理流媒体输入，几何-语义融合模块则通过语义指导解决几何噪声和语义模糊，点云优化模块则进行流形距离优化。

关键创新：最重要的创新在于引入了几何-语义融合机制和基于流形距离的点云优化策略。这些创新使得模型在处理噪声和边界划分方面表现优异，区别于传统方法。

关键设计：在设计中，采用了局部流形图进行点到流形的优化，动态激活和更新历史实例以实现快速优化，确保了模型在实时应用中的高效性。

🖼️ 关键图片

fig_0
fig_1
fig_2

📊 实验亮点

实验结果表明，Stream3Dv2在公共数据集上的性能显著优于现有基线，尤其在基础开放词汇流媒体3D分割和检测任务中，提升幅度达到XX%。这一成果展示了其在实际应用中的巨大潜力。

🎯 应用场景

该研究的潜在应用领域包括自动驾驶、机器人导航和增强现实等场景，能够在复杂环境中实现高效的3D场景理解。未来，该框架有望推动开放世界的智能体发展，提升人机交互的智能化水平。

📄 摘要（原文）

Recently, open-vocabulary zero-shot 3D scene understanding using vision foundation models has emerged as a promising alternative to data-intensive supervised methods. However, deploying these models in real-world scenarios is severely hindered by their inability to efficiently handle streaming RGB-D inputs and their inherent vulnerability to noise 2D segmentation masks. To address these critical limitations, we propose Stream3Dv2, a novel training-free framework designed for robust streaming 3D perception. Stream3Dv2 processes sequential data through an original nested local-to-historical architecture, capturing multi-view consistency while circumventing the high computational overhead so as to support timely responses. At its core, we introduce a comprehensive geometric-semantic fusion mechanism that resolves geometric noise and semantic ambiguity by explicitly utilizing semantic guidance and formulating 3D segmentation as solving point-and-set merging and partitioning problems. Furthermore, we present an innovative manifold-distance-based point cloud refinement strategy. This approach leverages local manifold graphs for point-to-manifold optimization that mitigates the boundary delineation failures caused by Euclidean-distance metrics, and employs geometric bounding boxes to dynamically activate and update historical instances for achieving rapid manifold-to-manifold refinement. Extensive experiments on public datasets demonstrate that Stream3Dv2 consistently outperforms existing baselines in foundational open-vocabulary streaming 3D segmentation and detection. Finally, we show that integrating our framework with an LLM-based agent enables advanced language-driven 3D scene understanding, underscoring its potential for open-world embodied intelligence. Code will be updated at https://github.com/SubmissionsIn/Stream3D.