GaussianObject: High-Quality 3D Object Reconstruction from Four Views with Gaussian Splatting

📄 arXiv: 2402.10259v4 📥 PDF

作者: Chen Yang, Sikuang Li, Jiemin Fang, Ruofan Liang, Lingxi Xie, Xiaopeng Zhang, Wei Shen, Qi Tian

分类: cs.CV, cs.GR

发布日期: 2024-02-15 (更新: 2024-11-13)

备注: ACM Transactions on Graphics (SIGGRAPH Asia 2024). Project page: https://gaussianobject.github.io/ Code: https://github.com/chensjtu/GaussianObject

🔗 代码/项目: GITHUB

💡 一句话要点

提出GaussianObject以解决稀疏视图下的3D重建问题

🎯 匹配领域: 支柱三：空间感知与语义 (Perception & Semantics)

关键词: 3D重建 高斯点云 扩散模型 多视图一致性 计算机视觉 虚拟现实 增强现实

📋 核心要点

  1. 现有方法在稀疏视图下重建3D对象面临多视图一致性和信息缺失的挑战。
  2. GaussianObject框架通过高斯点云表示和扩散模型修复，解决了3D重建中的信息不足问题。
  3. 在多个数据集上，GaussianObject仅用四个视图实现了优越的重建效果，超越了现有方法。

📝 摘要（中文）

从高度稀疏的视图重建和渲染3D对象对于推动3D视觉技术的应用和提升用户体验至关重要。然而，稀疏视图中的图像仅包含有限的3D信息，导致多视图一致性构建困难和对象信息缺失等挑战。为此，我们提出了GaussianObject框架，通过高斯点云表示和渲染3D对象，仅需4张输入图像即可实现高质量渲染。我们引入了视觉外壳和浮动物消除技术，帮助建立多视图一致性，并构建了基于扩散模型的高斯修复模型来补充缺失的信息。实验结果表明，GaussianObject在多个挑战性数据集上表现优异，显著超越了现有的最先进方法。

🔬 方法详解

问题定义：论文要解决从稀疏视图重建高质量3D对象的问题。现有方法在视图数量不足时，难以建立多视图一致性，且常常导致对象信息的部分缺失或压缩。

核心思路：论文提出的GaussianObject框架通过高斯点云表示3D对象，并利用扩散模型修复缺失信息，从而实现高质量的3D重建。这样的设计使得在仅有四张输入图像的情况下，仍能有效补充和优化对象信息。

技术框架：GaussianObject的整体架构包括两个主要阶段：首先，通过视觉外壳和浮动物消除技术建立初步的多视图一致性，生成粗略的3D高斯表示；其次，利用扩散模型构建高斯修复模型，进一步优化和补充对象信息。

关键创新：最重要的技术创新在于引入了高斯点云表示和基于扩散模型的修复策略，这与传统方法依赖于密集视图的重建方式有本质区别。

关键设计：在模型训练中，设计了自生成策略以获取训练所需的图像对，并且提出了COLMAP-free变体，使得不再依赖于预先给定的相机姿态，提升了模型的适用性和灵活性。

🖼️ 关键图片

fig_0
fig_1
fig_2

📊 实验亮点

在多个挑战性数据集（如MipNeRF360、OmniObject3D等）上，GaussianObject仅使用四个视图便实现了显著的性能提升，超越了现有的最先进方法，展现出其在3D重建领域的强大能力和应用前景。

🎯 应用场景

GaussianObject的研究成果在多个领域具有广泛的应用潜力，包括虚拟现实、增强现实、游戏开发以及工业设计等。通过高质量的3D重建，用户能够获得更真实的视觉体验，进而推动相关技术的进步和应用普及。

📄 摘要（原文）

Reconstructing and rendering 3D objects from highly sparse views is of critical importance for promoting applications of 3D vision techniques and improving user experience. However, images from sparse views only contain very limited 3D information, leading to two significant challenges: 1) Difficulty in building multi-view consistency as images for matching are too few; 2) Partially omitted or highly compressed object information as view coverage is insufficient. To tackle these challenges, we propose GaussianObject, a framework to represent and render the 3D object with Gaussian splatting that achieves high rendering quality with only 4 input images. We first introduce techniques of visual hull and floater elimination, which explicitly inject structure priors into the initial optimization process to help build multi-view consistency, yielding a coarse 3D Gaussian representation. Then we construct a Gaussian repair model based on diffusion models to supplement the omitted object information, where Gaussians are further refined. We design a self-generating strategy to obtain image pairs for training the repair model. We further design a COLMAP-free variant, where pre-given accurate camera poses are not required, which achieves competitive quality and facilitates wider applications. GaussianObject is evaluated on several challenging datasets, including MipNeRF360, OmniObject3D, OpenIllumination, and our-collected unposed images, achieving superior performance from only four views and significantly outperforming previous SOTA methods. Our demo is available at https://gaussianobject.github.io/, and the code has been released at https://github.com/GaussianObject/GaussianObject.