Find n' Propagate: Open-Vocabulary 3D Object Detection in Urban Environments
作者: Djamahl Etchegaray, Zi Huang, Tatsuya Harada, Yadan Luo
分类: cs.CV, cs.AI
发布日期: 2024-03-20 (更新: 2024-07-12)
备注: To appear in ECCV 2024. Source code: https://github.com/djamahl99/findnpropagate
🔗 代码/项目: GITHUB
💡 一句话要点
提出Find n' Propagate以解决城市环境中开放词汇3D物体检测问题
🎯 匹配领域: 支柱三：空间感知与语义 (Perception & Semantics)
关键词: 开放词汇学习 3D物体检测 LiDAR 城市环境 视觉-语言模型 自我训练 贪婪框搜索 远程模拟器
📋 核心要点
- 现有的LiDAR-based 3D物体检测系统受限于类别词汇，且新物体类别标注成本高，导致检测能力不足。
- 提出Find n' Propagate方法，通过贪婪框搜索器和远程模拟器等技术，提升新物体的检测能力和召回率。
- 实验结果表明，在多种开放词汇设置中，新物体召回率提高53%，平均精度提升达3.97倍。
📝 摘要（中文）
本文针对当前基于LiDAR的3D物体检测系统的局限性，特别是受限的类别词汇和新物体类别标注成本高的问题，探索了城市环境中的开放词汇学习。我们设计并基准测试了四种潜在解决方案，分为自上而下和自下而上的方法。为克服现有方法在3D框估计中遗漏新物体的局限性，提出了通用的Find n' Propagate方法，旨在最大化新物体的召回率，并将检测能力传播到更远的区域。通过贪婪框搜索器和远程模拟器等技术，显著提高了新物体的召回率和平均精度，实验结果显示在多种开放词汇设置中，召回率提高了53%。
🔬 方法详解
问题定义：本文旨在解决当前LiDAR-based 3D物体检测系统在开放词汇学习中的局限性，特别是新物体类别的检测和召回率不足的问题。现有方法往往对新物体的检测能力不足，且存在对相机近处物体的偏见。
核心思路：我们提出的Find n' Propagate方法通过贪婪框搜索器和远程模拟器，旨在最大化新物体的召回率，并将检测能力扩展到更远的区域。该方法通过多传感器数据和预训练的视觉-语言模型，增强了对新物体的识别能力。
技术框架：整体架构包括数据输入、贪婪框搜索、交叉对齐、密度排序和远程模拟器等模块。首先，输入多传感器数据，然后通过贪婪框搜索器识别新物体，接着通过交叉对齐和密度排序确保识别框的可靠性，最后利用远程模拟器进行自我训练。
关键创新：最重要的创新在于引入了远程模拟器，通过随机多样化伪标记的新实例，克服了对相机近处物体的偏见。这一设计使得检测能力得以扩展，显著提高了新物体的召回率。
关键设计：在技术细节上，采用了贪婪框搜索器来处理不同方向和深度的3D新框，并结合了密度排序算法以确保框的可靠性。此外，远程模拟器在自我训练过程中引入了多样性，增强了模型的泛化能力。
🖼️ 关键图片
📊 实验亮点
实验结果显示，Find n' Propagate方法在多种开放词汇设置中实现了53%的新物体召回率提升，并且在新物体类别的平均精度上达到了3.97倍的显著提高，展示了该方法的有效性和优越性。
🎯 应用场景
该研究在城市环境中的开放词汇3D物体检测具有广泛的应用潜力，尤其是在自动驾驶、智能交通系统和城市监控等领域。通过提高新物体的检测能力，能够有效提升系统的安全性和可靠性，推动智能交通技术的发展。
📄 摘要（原文）
In this work, we tackle the limitations of current LiDAR-based 3D object detection systems, which are hindered by a restricted class vocabulary and the high costs associated with annotating new object classes. Our exploration of open-vocabulary (OV) learning in urban environments aims to capture novel instances using pre-trained vision-language models (VLMs) with multi-sensor data. We design and benchmark a set of four potential solutions as baselines, categorizing them into either top-down or bottom-up approaches based on their input data strategies. While effective, these methods exhibit certain limitations, such as missing novel objects in 3D box estimation or applying rigorous priors, leading to biases towards objects near the camera or of rectangular geometries. To overcome these limitations, we introduce a universal \textsc{Find n' Propagate} approach for 3D OV tasks, aimed at maximizing the recall of novel objects and propagating this detection capability to more distant areas thereby progressively capturing more. In particular, we utilize a greedy box seeker to search against 3D novel boxes of varying orientations and depth in each generated frustum and ensure the reliability of newly identified boxes by cross alignment and density ranker. Additionally, the inherent bias towards camera-proximal objects is alleviated by the proposed remote simulator, which randomly diversifies pseudo-labeled novel instances in the self-training process, combined with the fusion of base samples in the memory bank. Extensive experiments demonstrate a 53% improvement in novel recall across diverse OV settings, VLMs, and 3D detectors. Notably, we achieve up to a 3.97-fold increase in Average Precision (AP) for novel object classes. The source code is made available at https://github.com/djamahl99/findnpropagate.