DivaTrack: Diverse Bodies and Motions from Acceleration-Enhanced Three-Point Trackers

📄 arXiv: 2402.09211v1 📥 PDF

作者: Dongseok Yang, Jiho Kang, Lingni Ma, Joseph Greer, Yuting Ye, Sung-Hee Lee

分类: cs.CV, cs.AI

发布日期: 2024-02-14

备注: accepted to Eurographics 2024

💡 一句话要点

提出DivaTrack以解决全身姿态推断问题

🎯 匹配领域: 支柱一：机器人控制 (Robot Control)

关键词: 全身姿态推断 深度学习 虚拟现实 运动捕捉 惯性测量单元 实时跟踪 多样化运动 姿态融合

📋 核心要点

  1. 现有三点跟踪方法在全身姿态推断中面临高度不确定性，难以适应不同体型和运动场景。
  2. DivaTrack通过结合IMU数据和两阶段模型，增强了姿态推断的准确性和稳定性。
  3. 在包含22名受试者的复杂运动数据集上，DivaTrack在实时跟踪精度上显著优于现有方法。

📝 摘要（中文）

全身虚拟化身在数字现实中的社交和环境交互中至关重要。然而，现有设备仅提供来自头戴设备和两个控制器的三点跟踪，导致全身姿态推断面临挑战。本文提出了一种深度学习框架DivaTrack，通过增强三点输入的稀疏性，结合惯性测量单元（IMU）的线性加速度，改善足部接触预测，并在两阶段模型中结合下半身和上半身姿态的预测，从而稳定推断的全身姿态。实验表明，该方法在实时跟踪用户多样化运动时表现优异。

🔬 方法详解

问题定义：本文旨在解决从三点跟踪器推断全身姿态的困难，现有方法由于输入稀疏性和不确定性，难以适应多样化的体型和运动场景。

核心思路：DivaTrack通过引入IMU的线性加速度数据，增强了对足部接触的预测，从而为下半身姿态提供更准确的条件，同时结合上半身姿态的预测，形成两阶段模型。

技术框架：整体架构包括数据输入模块（IMU和三点跟踪器）、足部接触预测模块、下半身和上半身姿态推断模块，以及姿态融合模块，确保在不同运动类型下的稳定性。

关键创新：DivaTrack的核心创新在于通过IMU数据增强姿态推断的准确性，并在两种参考框架下进行姿态融合，显著提升了对复杂运动的适应能力。

关键设计：在模型设计中，采用了特定的损失函数来优化足部接触预测，并设计了适应不同运动类型的网络结构，以提高整体推断的稳定性和准确性。

🖼️ 关键图片

fig_0
fig_1
fig_2

📊 实验亮点

在实验中，DivaTrack在实时跟踪用户多样化运动时表现出色，尤其在复杂运动（如弓步、呼啦圈和坐下）中，准确率显著高于现有方法，展示了其在多种场景下的有效性和稳定性。

🎯 应用场景

DivaTrack的研究成果在虚拟现实、增强现实和游戏等领域具有广泛的应用潜力。通过提供更准确的全身姿态跟踪，能够提升用户的沉浸感和交互体验，推动社交平台和运动模拟等应用的发展。

📄 摘要（原文）

Full-body avatar presence is crucial for immersive social and environmental interactions in digital reality. However, current devices only provide three six degrees of freedom (DOF) poses from the headset and two controllers (i.e. three-point trackers). Because it is a highly under-constrained problem, inferring full-body pose from these inputs is challenging, especially when supporting the full range of body proportions and use cases represented by the general population. In this paper, we propose a deep learning framework, DivaTrack, which outperforms existing methods when applied to diverse body sizes and activities. We augment the sparse three-point inputs with linear accelerations from Inertial Measurement Units (IMU) to improve foot contact prediction. We then condition the otherwise ambiguous lower-body pose with the predictions of foot contact and upper-body pose in a two-stage model. We further stabilize the inferred full-body pose in a wide range of configurations by learning to blend predictions that are computed in two reference frames, each of which is designed for different types of motions. We demonstrate the effectiveness of our design on a large dataset that captures 22 subjects performing challenging locomotion for three-point tracking, including lunges, hula-hooping, and sitting. As shown in a live demo using the Meta VR headset and Xsens IMUs, our method runs in real-time while accurately tracking a user's motion when they perform a diverse set of movements.