RL$^2$-VLA: Adaptive RL Latent Compositional Steering with Test-Time Scaling for Vision-Language-Action Models

📄 arXiv: 2607.26991v1 📥 PDF

作者: Derek Ming Siang Tan, Shailesh Shailesh, Srikrishna Iyer, William Wei Jie Teo, Yuanliang Ju, Qiao Gu, Guillaume Sartoretti

分类: cs.RO

发布日期: 2026-07-29

备注: Code and models are available at https://rl2-vla.github.io

💡 一句话要点

提出RL²以解决VLA模型在复杂任务中的性能下降问题

🎯 匹配领域: 支柱二：RL算法与架构 (RL & Architecture) 支柱九：具身大模型 (Embodied Foundation Models)

关键词: 视觉-语言-动作 强化学习 自适应引导 潜变量 机器人控制 多样性策略 超出领域任务 性能提升

📋 核心要点

  1. 现有的VLA模型在复杂和超出领域的任务中表现不佳，且现有的干预策略未能有效应对不同的成功与失败状态。
  2. RL²通过自适应推理时间引导，结合离线强化学习和VLA潜变量，优化了行动策略，提升了行动多样性。
  3. 在SIMPLER和PolaRiS基准测试中，RL²在超出领域设置中成功率提高了17.3%，并在真实世界实验中验证了其有效性。

📝 摘要（中文）

尽管视觉-语言-动作（VLA）模型展现了令人印象深刻的视觉运动能力，但在复杂和超出领域的任务中，其性能往往会下降。近期的测试时间引导和缩放方法在无需大量数据收集和重新训练的情况下改善了性能，但行动样本往往集中在相似的行为上，因此继承了相关的失败模式。此外，现有方法在每个时间步应用相同的干预策略，而不考虑基础策略是否已经可能成功。为了解决这些局限性，本文提出了RL²，一个自适应推理时间引导框架，利用强化学习在VLA潜变量上进行操作。通过在推理过程中将轻量级的离线RL策略与冻结的VLA流速组合，RL²显著提高了在超出领域设置中的成功率。

🔬 方法详解

问题定义：本文旨在解决VLA模型在复杂任务中的性能下降问题，现有方法在应对不同状态时的干预策略缺乏灵活性，导致失败模式的继承。

核心思路：RL²框架通过在推理时结合离线强化学习策略与VLA潜变量，动态调整干预策略，仅在预测失败时激活组合引导，从而提升行动的多样性和成功率。

技术框架：RL²的整体架构包括离线RL策略训练模块和推理阶段的组合引导模块。离线RL策略基于VLA专家的潜变量进行训练，推理时将其流速与冻结的VLA流速组合。

关键创新：RL²的核心创新在于自适应引导策略的设计，能够根据当前状态动态调整干预策略，与现有方法在每个时间步应用固定策略的方式有本质区别。

关键设计：在RL²中，离线RL策略的训练依赖于丰富的潜变量表示，损失函数设计考虑了行动多样性与成功率的平衡，确保了模型在不同状态下的有效性。

🖼️ 关键图片

fig_0
fig_1
fig_2

📊 实验亮点

在SIMPLER和PolaRiS基准测试中，RL²在超出领域设置中的成功率提高了17.3%。通过消融实验和缩放研究，验证了潜变量表示和强化学习训练的重要性，且在真实世界实验中也展示了良好的迁移能力。

🎯 应用场景

RL²框架具有广泛的应用潜力，特别是在需要高精度和灵活性的机器人控制、自动驾驶和人机交互等领域。其自适应引导策略能够有效应对复杂环境中的不确定性，提升系统的整体性能和可靠性。未来，RL²有望推动VLA模型在实际应用中的普及与发展。

📄 摘要（原文）

Despite the impressive visuomotor capabilities enabled by Vision-Language-Action (VLA) models, their performance often degrades on challenging and out-of-domain tasks. Recent test-time steering and scaling methods improve performance without extensive data collection and retraining, but action samples often remain concentrated around similar behaviors and therefore inherit correlated failure modes. Moreover, existing methods apply the same intervention strategy at every timestep, regardless of whether the base policy is already likely to succeed. To address these limitations, we introduce $RL^2$, an adaptive inference-time steering framework that leverages Reinforcement Learning on VLA Latents. First, we train a lightweight offline RL policy conditioned on expressive latents extracted from the VLA action expert and compose its flow velocity with that of the frozen VLA during inference. This compositional steering strategy combines the behavioral priors of large-scale imitation learning with the action diversity induced by offline RL beyond dominant demonstration modes. We further discover that inference-time steering follows fundamentally different scaling laws under success and failure states, revealing that action diversity is most beneficial when the base VLA is likely to fail, but can unnecessarily perturb already-accurate actions when success is likely. Building on this insight, $RL^2$ activates compositional steering only when failure is predicted. Across the SIMPLER and PolaRiS benchmarks, $RL^2$ improves success rates by up to +17.3% in out-of-domain settings, while ablations and scaling studies demonstrate the importance of latent representations and RL training. Finally, real-world experiments demonstrate that these gains transfer beyond simulation, establishing $RL^2$ as a practical and modular steering framework for VLA deployment.