Should the Action Expert Train the VLM? Action-Token Supervision and Gradient Routing in Mixture-of-Transformer VLAs

📄 arXiv: 2611.05249v1 📥 PDF

作者: Asher James Hancock, Boyi Li, Yuxiao Chen, Shuhan Tan, Wenhao Ding, Thomas Tian, Yan Wang, Boris Ivanovic, Marco Pavone

分类: cs.RO

发布日期: 2026-11-03

💡 一句话要点

提出动作令牌监督与梯度路由以优化VLA模型训练

🎯 匹配领域: 支柱一：机器人控制 (Robot Control) 支柱二：RL算法与架构 (RL & Architecture) 支柱九：具身大模型 (Embodied Foundation Models)

关键词: 视觉-语言-动作 流匹配专家 动作令牌监督 梯度耦合 模型训练优化 自动驾驶 机器人操作

📋 核心要点

  1. 现有的VLA模型训练方法未能有效利用VLM的梯度信息，导致动作预测性能受限。
  2. 论文提出通过动作令牌监督和专家损失的梯度更新来优化VLM的训练，探索两者的相互作用。
  3. 实验结果表明，耦合在有动作令牌监督时表现良好，且在闭环驾驶和机器人操作中显著提升性能。

📝 摘要（中文）

视觉-语言-动作（VLA）模型越来越多地将预训练的视觉-语言模型（VLM）与流匹配专家相结合，以基于VLM的关键-值表示来调整其动作预测。然而，VLM并未针对动作专家的流匹配目标进行训练。本文探讨了通过反向传播专家损失的梯度（耦合）或通过VLM监督来预测离散动作令牌的两种信号。研究发现，梯度绝缘在有动作令牌监督时与耦合相当，而在没有监督时，耦合能将轨迹预测误差降低约25%。当两者共同训练时，耦合在闭环驾驶中提高了约17%的路线进展，且在机器人操作中也表现出类似的提升。研究结果挑战了将绝缘作为VLA训练默认方法的观点。

🔬 方法详解

问题定义：本文旨在解决VLA模型中VLM未针对动作专家流匹配目标进行训练的问题，现有方法通过绝缘策略阻止梯度传递，导致动作预测能力受限。

核心思路：论文提出了结合动作令牌监督与专家损失梯度更新的训练策略，旨在通过这两种信号来优化VLM的动作相关表示学习。

技术框架：整体架构包括VLM和流匹配专家两个主要模块，VLM负责处理视觉和语言信息，专家则基于VLM的输出进行动作预测。训练过程中，采用耦合和绝缘两种策略进行对比实验。

关键创新：最重要的技术创新在于明确了动作令牌监督与梯度耦合的相互作用，揭示了绝缘策略在不同训练设置下的有效性与局限性。

关键设计：在实验中，设置了不同的损失函数以平衡耦合与绝缘的影响，同时调整了动作令牌的监督强度，以评估其对模型性能的影响。具体的网络结构和参数设置在实验部分进行了详细描述。

🖼️ 关键图片

fig_0
fig_1
fig_2

📊 实验亮点

实验结果显示，在有动作令牌监督的情况下，耦合策略与绝缘策略的性能相当；而在没有监督时，耦合策略能将轨迹预测误差降低约25%。此外，耦合策略在闭环驾驶中提高了约17%的路线进展，机器人操作性能也得到了显著提升。

🎯 应用场景

该研究的潜在应用领域包括自动驾驶和机器人操作等需要高精度动作预测的场景。通过优化VLA模型的训练策略，可以提高这些系统的智能化水平和操作效率，推动相关技术的实际应用与发展。

📄 摘要（原文）

Vision-language-action (VLA) models increasingly pair a pre-trained vision-language model (VLM) with a flow-matching expert that conditions its action predictions on the VLM's key-value representations. Those representations must carry information useful for control, yet the VLM was never trained for the action expert's flow-matching objective. Two signals can address this: gradients backpropagated from the expert's loss (coupling), or VLM supervision to predict discrete tokens encoding the same target actions. Prominent training recipes forgo the first, blocking expert-to-VLM gradients (insulation) to protect the VLM's existing capabilities, while also supervising it on vision-language data and action tokens. Because vision-language co-training also aims to preserve those capabilities, the additional protection provided by insulation is unclear. At the same time, insulation removes a direct signal about which features continuous action prediction requires. On the other hand, action-token supervision offers an alternative way to learn action-relevant representations, which may explain why insulation succeeds in practice. We disentangle these factors in autonomous driving and manipulation by independently varying action-token supervision and whether the expert's loss updates the VLM. In open-loop driving, insulation is competitive with coupling when action tokens are supervised; without them, coupling reduces trajectory-prediction error by roughly 25%. When both components are trained jointly from the outset, coupling increases closed-loop driving route progress by approximately 17% with action-token supervision and 40% without it, and similarly improves robot manipulation performance under both settings. These findings show that the value of gradient insulation depends on the VLM's action supervision and the evaluation setting, challenging insulation as a default VLA training recipe.