MRIC: Model-Based Reinforcement-Imitation Learning with Mixture-of-Codebooks for Autonomous Driving Simulation

📄 arXiv: 2404.18464v1 📥 PDF

作者: Baotian He, Yibing Li

分类: cs.RO

发布日期: 2024-04-29

备注: This work has been submitted to the IEEE for possible publication

💡 一句话要点

提出MRIC框架以解决自主驾驶仿真中的多模态行为模拟问题

🎯 匹配领域: 支柱二：RL算法与架构 (RL & Architecture) 支柱八：物理动画 (Physics-based Animation)

关键词: 自主驾驶 仿真 强化学习 模仿学习 多模态行为 状态匹配 正则化 行为压缩

📋 核心要点

  1. 现有方法在自主驾驶仿真中面临多模态行为模拟的挑战，导致行为分布不均和训练不稳定。
  2. 论文提出MRIC框架，通过可微仿真和双策略正则化来优化策略学习，解决低密度区域的梯度问题。
  3. 实验结果表明，MRIC在多样性、行为现实性和分布现实性方面显著优于现有最先进基线，尤其在碰撞率和时间到碰撞等关键指标上有显著提升。

📝 摘要（中文）

准确模拟异构代理在各种场景中的多样化行为是自主驾驶仿真的基础。由于行为分布的多模态性、高维驾驶场景、分布转移和信息不完整等挑战，任务变得复杂。本文首先通过可微仿真实现状态匹配，提供有效的学习信号并优化策略的信用分配，揭示了梯度高速公路和代理间梯度路径的存在。然而，发现低密度区域存在梯度爆炸和弱监督问题。为此，应用双策略正则化来缩小函数空间。此外，考虑到多样性，异构代理的行为可以有效压缩为一系列原型向量进行检索。最终提出了基于模型的强化模仿学习框架MRIC，结合开放式模型正则化和基于模型的强化学习正则化，显著提升了多样性和行为现实性。

🔬 方法详解

问题定义：本文旨在解决自主驾驶仿真中异构代理行为的多模态模拟问题。现有方法在高维场景下容易出现梯度爆炸和弱监督，导致训练效果不佳。

核心思路：通过可微仿真实现状态匹配，提供有效的学习信号，并应用双策略正则化来缩小函数空间，从而提高训练的稳定性和效率。

技术框架：MRIC框架包括三个主要模块：可微仿真模块、双策略正则化模块和基于原型向量的行为压缩模块。可微仿真模块用于状态匹配，正则化模块用于优化策略，行为压缩模块则用于提高多样性。

关键创新：MRIC的主要创新在于引入开放式模型正则化和基于模型的强化学习正则化，结合动态乘子机制，有效消除正则化干扰，确保训练的稳定性和有效性。

关键设计：在损失函数设计上，采用了基于Minkowski距离的碰撞避免机制和基于投影的交通规则合规奖励，确保模型在复杂场景中的表现。

🖼️ 关键图片

fig_0
fig_1
fig_2

📊 实验亮点

实验结果显示，MRIC在多样性、行为现实性和分布现实性方面显著优于现有最先进的基线，尤其在碰撞率、最小SADE和时间到碰撞的JSD等关键指标上，提升幅度达到显著水平，验证了其有效性。

🎯 应用场景

该研究的潜在应用领域包括自动驾驶汽车的行为预测、交通仿真和智能交通系统的优化。通过提升自主驾驶系统的行为模拟能力，MRIC框架有望在实际应用中提高安全性和效率，推动智能交通的发展。

📄 摘要（原文）

Accurately simulating diverse behaviors of heterogeneous agents in various scenarios is fundamental to autonomous driving simulation. This task is challenging due to the multi-modality of behavior distribution, the high-dimensionality of driving scenarios, distribution shift, and incomplete information. Our first insight is to leverage state-matching through differentiable simulation to provide meaningful learning signals and achieve efficient credit assignment for the policy. This is demonstrated by revealing the existence of gradient highways and interagent gradient pathways. However, the issues of gradient explosion and weak supervision in low-density regions are discovered. Our second insight is that these issues can be addressed by applying dual policy regularizations to narrow the function space. Further considering diversity, our third insight is that the behaviors of heterogeneous agents in the dataset can be effectively compressed as a series of prototype vectors for retrieval. These lead to our model-based reinforcement-imitation learning framework with temporally abstracted mixture-of-codebooks (MRIC). MRIC introduces the open-loop modelbased imitation learning regularization to stabilize training, and modelbased reinforcement learning (RL) regularization to inject domain knowledge. The RL regularization involves differentiable Minkowskidifference-based collision avoidance and projection-based on-road and traffic rule compliance rewards. A dynamic multiplier mechanism is further proposed to eliminate the interference from the regularizations while ensuring their effectiveness. Experimental results using the largescale Waymo open motion dataset show that MRIC outperforms state-ofthe-art baselines on diversity, behavioral realism, and distributional realism, with large margins on some key metrics (e.g., collision rate, minSADE, and time-to-collision JSD).