LAWM-3D: Learning 3D-Aware Latent Actions from Human Videos for Generalizable Robot World Models

📄 arXiv: 2608.05706v1 📥 PDF

作者: Jiarui Yang, Jiale Zhange, Jiawei Li, Hang Guo, Wen Huang, Jinpeng Wang, Peidong Liu, Shu-Tao Xia

分类: cs.CV

发布日期: 2026-08-06

💡 一句话要点

提出LAWM-3D以解决机器人世界模型中的3D感知问题

🎯 匹配领域: 支柱二：RL算法与架构 (RL & Architecture) 支柱九：具身大模型 (Embodied Foundation Models)

关键词: 潜在动作模型 3D感知 多视角学习 几何对齐 机器人世界模型

📋 核心要点

  1. 现有的潜在动作模型在多视角输入下无法有效学习3D感知，主要受限于未来帧信息泄露和视角变化。
  2. 本文提出LAWM-3D，通过多视角不变的动作标记方案和几何对齐约束，解决了3D感知学习中的关键问题。
  3. 实验结果表明，LAWM-3D在生成质量、物理一致性和泛化能力上显著优于现有方法，达到了最新的性能标准。

📝 摘要（中文）

世界模型使得智能体能够在没有真实世界交互的情况下进行前向推理和规划。然而，现有方法在开放世界的应用受到高昂的动作标注成本和平台间动作空间异质性的限制。最近的潜在动作模型（LAMs）通过自监督方式直接从未标注的人类视频中学习动作表示，缓解了这一瓶颈。然而，大多数现有的LAMs依赖于单视角输入，主要在2D像素空间中操作。本文提出LAWM-3D，结合多视角视频训练，设计了多视角不变的统一动作标记方案、几何对齐约束和非单射RGB-D联合重建目标，显著提升了世界模型的性能，达到了生成质量、物理一致性和泛化能力的最新水平。

🔬 方法详解

问题定义：本文旨在解决现有潜在动作模型在多视角视频训练中无法有效学习3D感知的问题。现有方法主要依赖单视角输入，导致动作表示缺乏空间一致性和几何信息。

核心思路：LAWM-3D通过引入多视角不变的统一动作标记方案和几何对齐约束，确保学习到的潜在动作具有3D感知能力，从而提升机器人在复杂环境中的表现。

技术框架：该方法采用两阶段的训练流程，首先进行大规模人类视频的预训练，然后进行机器人任务的微调。主要模块包括多视角动作标记、几何对齐和RGB-D重建。

关键创新：LAWM-3D的核心创新在于将多视角输入与几何对齐相结合，克服了现有方法在3D感知方面的局限性，确保了动作学习的几何一致性。

关键设计：在设计中，采用了多视角不变的动作标记方案，几何对齐约束将中间特征与预训练的3D基础模型对齐，以及非单射RGB-D联合重建目标，以防止未来帧信息的泄露。具体的损失函数和网络结构设计确保了模型的有效性。

🖼️ 关键图片

fig_0
fig_1
fig_2

📊 实验亮点

实验结果显示，LAWM-3D在生成质量、物理一致性和泛化能力上均达到了最新的性能标准，相较于基线方法，生成质量提升了XX%，物理一致性提升了YY%，泛化能力提升了ZZ%。这些结果表明，LAWM-3D在实际应用中具有显著优势。

🎯 应用场景

该研究的潜在应用领域包括机器人导航、自动驾驶、虚拟现实和增强现实等。通过提升机器人对环境的理解能力，LAWM-3D能够在复杂场景中实现更为精准的动作规划和执行，推动智能体在开放世界中的应用。未来，该技术可能会影响人机交互、智能家居和自主系统等多个领域。

📄 摘要（原文）

World models enable agents to perform forward rollout and planning without real-world interaction. However, their application in open-world embodied intelligence remains limited by the high cost of action annotations and the heterogeneity of action spaces across platforms. Recently, latent action models (LAMs) have alleviated this bottleneck by learning action representations directly from unlabeled human videos in a self-supervised manner. Nevertheless, most existing LAMs rely on single-view inputs and operate primarily in 2D pixel space, raising a fundamental question: can simply incorporating multi-view videos into LAM training endow the learned latent actions with 3D-aware perception? Our study shows that the answer is negative. The primary reasons lie in future-frame appearance leakage as well as inter-camera appearance discrepancies and viewpoint variations. To address these issues, we propose LAWM-3D, which introduces three tightly coupled key designs: (1) a multi-view invariant unified action tokenization scheme for learning 3D-aware latent actions; (2) a geometric alignment constraint that anchors intermediate encoder features to a pretrained 3D foundation model, thereby explicitly providing cross-view geometric correspondences; and (3) a non-injective RGB-D joint reconstruction objective that prevents shortcut learning from future-frame appearance information, forcing the LAM to focus supervision on motion cues with geometric significance. Importantly, these components are not simply stacked but are tightly coupled through a unified motivation. Built upon a two-stage paradigm of large-scale human video pretraining followed by robot fine-tuning, extensive experiments demonstrate that the proposed 3D-aware latent actions significantly improve world model performance, achieving SOTA results in generation quality, physical consistency, and generalization ability.