PSAvatar: A Point-based Shape Model for Real-Time Head Avatar Animation with 3D Gaussian Splatting

📄 arXiv: 2401.12900v5 📥 PDF

作者: Zhongyuan Zhao, Zhenyu Bao, Qing Li, Guoping Qiu, Kanglin Liu

分类: cs.GR, cs.CV

发布日期: 2024-01-23 (更新: 2024-06-24)

备注: 13 pages, 10 figures

💡 一句话要点

提出PSAvatar以解决实时高保真头部虚拟形象动画问题

🎯 匹配领域: 支柱三：空间感知与语义 (Perception & Semantics)

关键词: 头部虚拟形象 实时动画 3D高斯 参数化模型 计算机视觉 虚拟现实 细节表示

📋 核心要点

  1. 现有方法在实时高保真头部虚拟形象动画中面临速度与质量的权衡，且难以建模复杂的非面部结构。
  2. 本文提出PSAvatar框架，利用点基的可变形形状模型和3D高斯实现高保真细节表示，克服了现有方法的不足。
  3. 实验结果显示，PSAvatar能够在512×512分辨率下以≥25帧每秒的速度实时动画化多种高保真头部虚拟形象。

📝 摘要（中文）

尽管已有显著进展，实现实时高保真的头部虚拟形象动画仍然困难，现有方法在速度与质量之间存在权衡。基于3DMM的方法通常无法有效建模眼镜和发型等非面部结构，而神经隐式模型则面临变形灵活性不足和渲染效率低下的问题。本文提出PSAvatar，一个新颖的框架，利用离散几何原语创建参数化可变形形状模型，并采用3D高斯进行细节表示和高保真渲染。该模型通过点而非网格进行3D表示，增强了表示灵活性，能够重建多种复杂几何形状。实验表明，PSAvatar能够以每秒≥25帧的速度实时动画化高保真头部虚拟形象。

🔬 方法详解

问题定义：本文旨在解决实时高保真头部虚拟形象动画中的速度与质量之间的矛盾，现有方法在建模复杂几何形状时存在显著不足。

核心思路：PSAvatar通过引入参数化可变形形状模型（PMSM）和3D高斯，利用离散几何原语来增强表示灵活性，从而实现高保真细节的渲染。

技术框架：该框架主要包括两个模块：首先将FLAME网格转换为点云，接着通过分析-合成的方式对齐这些点，以便利用3D高斯进行细节表示和外观建模。

关键创新：PSAvatar的核心创新在于使用点而非传统网格进行3D表示，允许更灵活的形状变换和细节重建，显著提升了对复杂结构的建模能力。

关键设计：在模型设计中，采用了特定的点采样策略以确保对复杂几何形状的有效重建，同时结合了适当的损失函数以优化细节表示和渲染效果。

🖼️ 关键图片

fig_0
fig_1
fig_2

📊 实验亮点

实验结果表明，PSAvatar能够在512×512分辨率下以≥25帧每秒的速度实时生成高保真的头部虚拟形象，相较于现有方法在动画流畅性和细节表现上有显著提升，展示了其在实时应用中的优越性。

🎯 应用场景

PSAvatar的研究成果在虚拟现实、游戏开发和在线社交平台等领域具有广泛的应用潜力。通过实现高保真的实时头部虚拟形象动画，该技术可以提升用户体验，促进虚拟形象的个性化和互动性，未来可能在数字人类和虚拟助手等应用中发挥重要作用。

📄 摘要（原文）

Despite much progress, achieving real-time high-fidelity head avatar animation is still difficult and existing methods have to trade-off between speed and quality. 3DMM based methods often fail to model non-facial structures such as eyeglasses and hairstyles, while neural implicit models suffer from deformation inflexibility and rendering inefficiency. Although 3D Gaussian has been demonstrated to possess promising capability for geometry representation and radiance field reconstruction, applying 3D Gaussian in head avatar creation remains a major challenge since it is difficult for 3D Gaussian to model the head shape variations caused by changing poses and expressions. In this paper, we introduce PSAvatar, a novel framework for animatable head avatar creation that utilizes discrete geometric primitive to create a parametric morphable shape model and employs 3D Gaussian for fine detail representation and high fidelity rendering. The parametric morphable shape model is a Point-based Morphable Shape Model (PMSM) which uses points instead of meshes for 3D representation to achieve enhanced representation flexibility. The PMSM first converts the FLAME mesh to points by sampling on the surfaces as well as off the meshes to enable the reconstruction of not only surface-like structures but also complex geometries such as eyeglasses and hairstyles. By aligning these points with the head shape in an analysis-by-synthesis manner, the PMSM makes it possible to utilize 3D Gaussian for fine detail representation and appearance modeling, thus enabling the creation of high-fidelity avatars. We show that PSAvatar can reconstruct high-fidelity head avatars of a variety of subjects and the avatars can be animated in real-time ($\ge$ 25 fps at a resolution of 512 $\times$ 512 ).