Aligning Multi-Trajectory Supervision with Policy Optimization for VLA Driving

📄 arXiv: 2608.30122v1 📥 PDF

作者: Tian Zhang, Zhuo Huang, Hongrui Ye, Yu Wu, Zengmao Wang, Kaixuan Zhou

分类: cs.CV, cs.AI, cs.LG

发布日期: 2026-08-31

💡 一句话要点

提出多轨迹监督与策略优化对齐框架以提升VLA驾驶性能

🎯 匹配领域: 支柱二：RL算法与架构 (RL & Architecture) 支柱九：具身大模型 (Embodied Foundation Models)

关键词: 视觉-语言-行动 多轨迹模仿学习 策略优化 帕累托最优 动态蒸馏 智能驾驶 机器人导航

📋 核心要点

  1. 现有的VLA驾驶方法在多轨迹模仿学习与GRPO结合时，轨迹选择对性能影响显著，但高评分轨迹可能导致策略偏差。
  2. 本文提出了一种新框架，通过将多轨迹监督与策略优化对齐，采用帕累托最优标准和可行性优先机制来改善策略更新。
  3. 在NAVSIM v1和v2上，所提方法在单轨迹推理下分别达到了91.4 PDMS和89.1 EPDMS，恢复了440个失败场景，提升幅度达到11.1%。

📝 摘要（中文）

随着视觉-语言-行动（VLA）驾驶方法的进步，越来越多地结合了多轨迹模仿学习与群体相对策略优化（GRPO），使得轨迹选择对最终性能至关重要。然而，一些高评分轨迹虽然能改善模仿，但可能会通过引导优势估计与当前策略的可行行为分布不对齐，从而降低后续GRPO的效果。为此，本文提出了一种新框架，将多轨迹监督与策略优化对齐。该框架通过将增强轨迹约束在真实可行区域的邻近流形上，并采用帕累托最优标准来替代传统的聚合评分，从源头过滤掉冲突样本。为确保扩展的轨迹监督在策略优化中有效吸收，本文引入了可行性优先的优势分配和动态蒸馏两种互补机制。实验结果表明，该方法在NAVSIM v1和v2上分别达到了91.4 PDMS和89.1 EPDMS的性能，显著提升了场景恢复能力。

🔬 方法详解

问题定义：本文旨在解决VLA驾驶中多轨迹模仿学习与策略优化之间的对齐问题，现有方法中高评分轨迹可能导致策略更新偏离安全行为。

核心思路：提出的框架通过约束增强轨迹在真实可行区域的邻近流形上，并采用帕累托最优标准来过滤冲突样本，从而实现多轨迹监督与策略优化的有效对齐。

技术框架：整体架构包括轨迹选择、优势分配和动态蒸馏三个主要模块。轨迹选择阶段通过帕累托标准筛选轨迹，优势分配阶段确保可行性优先，动态蒸馏则在多个迭代中更新教师轨迹。

关键创新：最重要的创新在于引入了可行性优先的优势分配机制和动态蒸馏策略，这与传统方法的聚合评分方式有本质区别，能够有效过滤不合适的轨迹。

关键设计：在参数设置上，采用了基于可行性组成的帕累托信用分配，损失函数设计上强调可行性优先，网络结构则支持动态更新教师轨迹以持续传递有效监督。

🖼️ 关键图片

fig_0
fig_1
fig_2

📊 实验亮点

实验结果显示，所提方法在NAVSIM v1和v2上分别达到了91.4 PDMS和89.1 EPDMS的性能，显著优于原始GRPO基线，恢复了440个失败场景，提升幅度达到11.1%。

🎯 应用场景

该研究的潜在应用领域包括自动驾驶、智能交通系统和机器人导航等。通过提升VLA驾驶的策略优化能力，能够在复杂环境中实现更安全、更高效的自动驾驶解决方案，具有重要的实际价值和未来影响。

📄 摘要（原文）

Vision-language-action (VLA) driving methods increasingly combine multi-trajectory imitation learning with group-relative policy optimization (GRPO), making trajectory selection critical to final performance. However, some high-scoring trajectories that improve imitation can degrade subsequent GRPO by inducing advantage estimates misaligned with the current policy's feasible behavior distribution, driving updates away from safe and compliant behaviors. To address this, we propose a novel framework that aligns multi-trajectory supervision with policy optimization. To address the policy gradient bias induced by infeasible noisy trajectories outside the feasible region, augmented trajectories are constrained to a neighboring manifold of the ground-truth feasible region, and a Pareto-optimality criterion is adopted in place of the conventional aggregate score, retaining only non-dominated candidates and thereby filtering out conflicting samples at the source. To ensure that expanded trajectory supervision is effectively absorbed during policy optimization, we introduce two complementary mechanisms: feasibility-first advantage assignment and dynamic distillation. The former adapts Pareto credit to the feasibility composition of each rollout group and guides fully infeasible groups toward safe references. The latter updates teacher trajectories across refinement rounds to continually transfer useful supervision. Together, they progressively translate the benefits of expanded supervision into policy improvement. On NAVSIM v1 and v2, our method achieves 91.4 PDMS and 89.1 EPDMS, respectively, under single-trajectory inference, and recovers 440 of 658 initially failed scenes, 11.1\% higher than the original GRPO baseline.