MetaDreamer: Efficient Text-to-3D Creation With Disentangling Geometry and Texture

📄 arXiv: 2311.10123v1 📥 PDF

作者: Lincong Feng, Muyu Wang, Maoyu Wang, Kuo Xu, Xiaoli Liu

分类: cs.CV

发布日期: 2023-11-16

备注: arXiv admin note: text overlap with arXiv:2306.17843, arXiv:2209.14988 by other authors

💡 一句话要点

提出MetaDreamer以解决3D生成中的几何与纹理纠缠问题

🎯 匹配领域: 支柱二：RL算法与架构 (RL & Architecture)

关键词: 3D生成 文本到3D 几何优化 纹理优化 生成模型 多视角一致性 图像控制

📋 核心要点

  1. 现有3D生成方法存在几何与纹理之间的纠缠问题，导致多视角不一致和生成速度慢。
  2. MetaDreamer采用两阶段优化策略，首先优化几何表示，然后细化几何和纹理，减轻相互依赖。
  3. 实验表明，MetaDreamer在20分钟内生成高质量3D物体，效率和质量均处于当前最优水平。

📝 摘要（中文）

生成模型在3D物体合成方面取得了显著进展，尤其是结合了从2D扩散模型中提取的先验知识。然而，现有3D合成框架仍面临多视角几何不一致和生成速度慢等挑战。为此，本文提出MetaDreamer，一种两阶段优化方法，分别优化几何表示和纹理，从而有效减轻几何与纹理之间的相互依赖。MetaDreamer能够在20分钟内基于文本提示生成高质量的3D物体，且在效率上处于当前文本到3D生成方法的前沿。此外，本文还引入了图像控制，增强了3D生成的可控性。

🔬 方法详解

问题定义：本文旨在解决现有3D生成方法中几何与纹理纠缠导致的多视角几何不一致和生成速度慢的问题。现有方法缺乏丰富的几何先验知识，影响了优化效果。

核心思路：MetaDreamer通过两阶段优化策略，首先专注于几何表示的优化，确保多视角一致性；其次，细化几何并优化纹理，从而实现更精细的3D物体生成。这样的设计旨在有效减轻几何与纹理之间的相互依赖。

技术框架：MetaDreamer的整体架构分为两个主要阶段：第一阶段优化几何表示，确保3D物体的准确性和一致性；第二阶段则集中于几何细化和纹理优化。每个阶段都有明确的优化目标，确保生成过程的高效性。

关键创新：MetaDreamer的主要创新在于其两阶段优化策略，清晰地分离了几何与纹理的优化过程。这一方法显著提高了生成效率，并在质量上达到了当前最先进的水平。

关键设计：在设计中，MetaDreamer采用了特定的损失函数来平衡几何与纹理的优化，同时利用丰富的2D和3D先验知识来指导优化过程。网络结构方面，采用了适合多视角生成的模块化设计，以提高生成的准确性和效率。

🖼️ 关键图片

fig_0
fig_1
fig_2

📊 实验亮点

MetaDreamer在20分钟内生成高质量3D物体，显著提高了生成效率。与现有方法相比，其在多视角一致性和生成速度上均表现出色，成为当前文本到3D生成领域的最优解。

🎯 应用场景

MetaDreamer在游戏开发、虚拟现实和增强现实等领域具有广泛的应用潜力。其高效的3D生成能力可以为内容创作者提供快速生成高质量3D资产的工具，推动相关行业的发展。此外，图像控制的引入使得用户可以更精确地控制生成结果，提升了用户体验。

📄 摘要（原文）

Generative models for 3D object synthesis have seen significant advancements with the incorporation of prior knowledge distilled from 2D diffusion models. Nevertheless, challenges persist in the form of multi-view geometric inconsistencies and slow generation speeds within the existing 3D synthesis frameworks. This can be attributed to two factors: firstly, the deficiency of abundant geometric a priori knowledge in optimization, and secondly, the entanglement issue between geometry and texture in conventional 3D generation methods.In response, we introduce MetaDreammer, a two-stage optimization approach that leverages rich 2D and 3D prior knowledge. In the first stage, our emphasis is on optimizing the geometric representation to ensure multi-view consistency and accuracy of 3D objects. In the second stage, we concentrate on fine-tuning the geometry and optimizing the texture, thereby achieving a more refined 3D object. Through leveraging 2D and 3D prior knowledge in two stages, respectively, we effectively mitigate the interdependence between geometry and texture. MetaDreamer establishes clear optimization objectives for each stage, resulting in significant time savings in the 3D generation process. Ultimately, MetaDreamer can generate high-quality 3D objects based on textual prompts within 20 minutes, and to the best of our knowledge, it is the most efficient text-to-3D generation method. Furthermore, we introduce image control into the process, enhancing the controllability of 3D generation. Extensive empirical evidence confirms that our method is not only highly efficient but also achieves a quality level that is at the forefront of current state-of-the-art 3D generation techniques.