DO3D: Self-supervised Learning of Decomposed Object-aware 3D Motion and Depth from Monocular Videos

📄 arXiv: 2403.05895v1 📥 PDF

作者: Xiuzhe Wu, Xiaoyang Lyu, Qihao Huang, Yong Liu, Yang Wu, Ying Shan, Xiaojuan Qi

分类: cs.CV

发布日期: 2024-03-09

备注: 24 pages, 14 figures, Tech Report

💡 一句话要点

提出DO3D以解决单目视频中动态物体的深度与运动估计问题

🎯 匹配领域: 支柱三：空间感知与语义 (Perception & Semantics) 支柱七：动作重定向 (Motion Retargeting) 支柱八：物理动画 (Physics-based Animation)

关键词: 自监督学习 深度估计 3D运动 动态场景 单目视频

📋 核心要点

  1. 现有方法将视频中的物体视为静态，无法有效建模动态场景中的物体运动，导致深度估计不准确。
  2. 本文提出的DO3D模块能够分解相机自运动和物体运动，协同深度估计，提升动态场景的建模能力。
  3. 在KITTI基准测试中，模型在高分辨率设置下的绝对相对深度误差为0.099，光流估计的整体EPE为7.09，均优于现有方法。

📝 摘要（中文）

尽管在单目视频的自监督深度估计方面取得了显著进展，但现有方法通常将视频中的所有物体视为静态实体，这违背了真实场景的动态特性，无法有效建模移动物体的几何和运动。本文提出了一种自监督方法，旨在从单目视频中联合学习3D运动和深度。该系统包含一个深度估计模块和一个新的分解物体感知3D运动（DO3D）估计模块，能够分别预测相机自运动和3D物体运动。深度和运动网络协同工作，真实建模场景的几何和动态，进而提升深度和3D运动的估计效果。实验结果表明，该模型在KITTI、Cityscapes和VKITTI2等基准数据集上表现优越，尤其在高分辨率设置下，绝对相对深度误差达到0.099。

🔬 方法详解

问题定义：本文旨在解决现有自监督深度估计方法无法有效处理动态物体运动的问题。现有方法通常将所有物体视为静态，导致对真实场景的几何和动态建模不足。

核心思路：提出DO3D模块，通过分解相机自运动和3D物体运动，分别进行建模，从而更准确地捕捉动态场景的特征。该设计使得深度和运动估计能够相互促进，提升整体性能。

技术框架：整体架构包括深度估计模块和DO3D估计模块。深度模块负责预测场景深度，而DO3D模块则负责分解并预测相机运动和物体运动。两者的预测结果结合生成新的视频帧用于自监督训练。

关键创新：DO3D模块是本文的核心创新，能够将非刚性3D物体运动分解为物体级的6自由度全局变换和像素级的局部3D运动变形场。这一设计显著提升了动态区域的运动估计能力。

关键设计：模型采用特定的损失函数来平衡深度和运动的估计，网络结构经过优化以适应动态场景的复杂性。具体参数设置和网络架构细节将在代码中提供。

🖼️ 关键图片

fig_0
fig_1
fig_2

📊 实验亮点

在KITTI基准测试中，本文模型在高分辨率设置下的绝对相对深度误差为0.099，优于所有对比研究。同时，光流估计的整体EPE为7.09，显著提升了动态区域的估计精度，展示了运动模型的有效性。

🎯 应用场景

该研究的潜在应用领域包括自动驾驶、机器人导航和增强现实等。通过准确的深度和运动估计，系统能够更好地理解和互动于动态环境，提升智能系统的自主决策能力。未来，该方法有望在更广泛的视觉理解任务中发挥重要作用。

📄 摘要（原文）

Although considerable advancements have been attained in self-supervised depth estimation from monocular videos, most existing methods often treat all objects in a video as static entities, which however violates the dynamic nature of real-world scenes and fails to model the geometry and motion of moving objects. In this paper, we propose a self-supervised method to jointly learn 3D motion and depth from monocular videos. Our system contains a depth estimation module to predict depth, and a new decomposed object-wise 3D motion (DO3D) estimation module to predict ego-motion and 3D object motion. Depth and motion networks work collaboratively to faithfully model the geometry and dynamics of real-world scenes, which, in turn, benefits both depth and 3D motion estimation. Their predictions are further combined to synthesize a novel video frame for self-supervised training. As a core component of our framework, DO3D is a new motion disentanglement module that learns to predict camera ego-motion and instance-aware 3D object motion separately. To alleviate the difficulties in estimating non-rigid 3D object motions, they are decomposed to object-wise 6-DoF global transformations and a pixel-wise local 3D motion deformation field. Qualitative and quantitative experiments are conducted on three benchmark datasets, including KITTI, Cityscapes, and VKITTI2, where our model delivers superior performance in all evaluated settings. For the depth estimation task, our model outperforms all compared research works in the high-resolution setting, attaining an absolute relative depth error (abs rel) of 0.099 on the KITTI benchmark. Besides, our optical flow estimation results (an overall EPE of 7.09 on KITTI) also surpass state-of-the-art methods and largely improve the estimation of dynamic regions, demonstrating the effectiveness of our motion model. Our code will be available.