$\nabla$Mimic: Ultra-Efficient Humanoid Control with Differentiable Simulation

📄 arXiv: 2610.41654v1 📥 PDF

作者: Weyl Lu, Yubei Chen

分类: cs.RO, cs.LG

发布日期: 2026-10-30

备注: 26 pages, 7 figures

💡 一句话要点

提出$ abla$Mimic以解决高效类人控制问题

🎯 匹配领域: 支柱一：机器人控制 (Robot Control) 支柱二：RL算法与架构 (RL & Architecture) 支柱八：物理动画 (Physics-based Animation)

关键词: 类人机器人 策略学习 可微分模拟 梯度裁剪 多步回归 高效控制 NablaEngine

📋 核心要点

  1. 现有的零阶方法如PPO在类人机器人控制中样本需求极高，导致效率低下。
  2. 论文提出的$ abla$Mimic通过中位数相对梯度裁剪和多步回归，解决了接触动态微分不稳定的问题。
  3. 在全身跟踪实验中，$ abla$Mimic在20分钟内达到了0.82的分数，远超传统方法的效率。

📝 摘要（中文）

机器人策略学习通常依赖于零阶方法如PPO，这些方法将模拟器视为黑箱，导致样本数量巨大。尽管一阶策略学习通过动力学进行微分，但在接触丰富的全身控制中效果不佳，梯度常常爆炸。本文提出$ abla$Mimic，通过中位数相对每样本梯度裁剪和多步回归，显著提高了类人运动跟踪的效率，并在GPU上实现了高效的可微分模拟引擎NablaEngine，最终在多个实验中展示了优越的性能。

🔬 方法详解

问题定义：本文旨在解决现有类人机器人控制中一阶策略学习的梯度爆炸问题，特别是在接触丰富的动态环境下，现有方法往往无法有效利用梯度信息。

核心思路：$ abla$Mimic通过引入中位数相对梯度裁剪，限制了不稳定的梯度影响，并通过多步回归重用模拟窗口，从而提高了策略学习的效率。

技术框架：整体架构包括NablaEngine，一个可微分的GPU模拟引擎，能够高效计算接触动态，并通过单个巨核实现完整控制步骤的反向传播。主要模块包括模拟器、梯度裁剪和多步回归。

关键创新：最重要的创新在于使用精确的梯度信息，而非近似的替代梯度，结合中位数相对梯度裁剪，显著提高了学习效率和稳定性。

关键设计：在设计中，采用了多步回归策略以独立于梯度规模限制每个动作目标，同时NablaEngine通过融合每个世界的内核，提升了计算效率，达到比现有MuJoCo Warp adjoint快11.5倍的性能。

📊 实验亮点

在全身跟踪实验中，$ abla$Mimic在20分钟内达到了0.82的分数，而传统的SHAC方法在几分钟内因梯度爆炸而停止。最终，$ abla$Mimic在1.8小时内达到了0.874的分数，展示了显著的效率提升。

🎯 应用场景

该研究的潜在应用领域包括类人机器人控制、自动化和人机交互等。通过提高类人机器人的控制效率，能够在更复杂的环境中实现更高效的任务执行，未来可能推动智能机器人在家庭、工业等领域的广泛应用。

📄 摘要（原文）

Robot policy learning is dominated by zeroth-order methods such as PPO, which treat the simulator as a black box and pay for it with enormous sample counts. First-order policy learning differentiates through the dynamics, but on contact-rich whole-body control it has rarely worked: its gradients explode. A recent world-model method showed that derivative information can make humanoid motion tracking far more efficient, using surrogate gradients of a learned model. We show that exact gradients of the simulator itself do better, once two obstacles are removed. Exact gradients contain rare unbounded samples, because contact dynamics is not differentiable everywhere and its derivative diverges near some events; and they are expensive, because the simulator's adjoint has to replay the contact solve. $\nabla$Mimic bounds these samples with median-relative per-sample gradient clipping at every step of the backward pass, and reuses each costly simulated window by multistep regression onto gradient-derived action targets, which bounds every action target independently of the gradient scale. NablaEngine, a differentiable GPU simulation engine for the Unitree G1, removes the cost: it computes MuJoCo's full contact dynamics with fused per-world kernels and the adjoint of a whole control step as a single megakernel, with analytic gradients that match float64 finite differences, $11.5\times$ faster than the available MuJoCo Warp adjoint. On whole-body tracking of all 40 LAFAN1 clips with one policy on one GPU, $\nabla$Mimic reaches the score reported by the state-of-the-art world-model method, 0.82, in 20 minutes instead of 1.2 h, and 0.874 in 1.8 h. SHAC, the standard first-order baseline, stops on exploding gradients within minutes. We release NablaEngine and $\nabla$Mimic as open-source infrastructure for first-order humanoid policy learning.