Diffusion-ES: Gradient-free Planning with Diffusion for Autonomous Driving and Zero-Shot Instruction Following
作者: Brian Yang, Huangyuan Su, Nikolaos Gkanatsios, Tsung-Wei Ke, Ayush Jain, Jeff Schneider, Katerina Fragkiadaki
分类: cs.LG, cs.AI, cs.CL, cs.RO
发布日期: 2024-02-09 (更新: 2024-07-16)
💡 一句话要点
提出Diffusion-ES以解决自主驾驶中的非可微优化问题
🎯 匹配领域: 支柱一：机器人控制 (Robot Control) 支柱九：具身大模型 (Embodied Foundation Models)
关键词: 自主驾驶 轨迹优化 扩散模型 无梯度优化 复杂决策 黑箱优化 进化搜索
📋 核心要点
- 现有的奖励梯度引导去噪方法需要可微的奖励函数，限制了其在非可微优化问题中的应用。
- Diffusion-ES结合了无梯度优化与轨迹去噪，能够在不依赖可微奖励函数的情况下优化复杂目标。
- Diffusion-ES在nuPlan基准测试中表现出色，超越了现有的规划方法，能够生成复杂的驾驶行为。
📝 摘要（中文）
扩散模型在复杂和多模态轨迹分布建模方面表现出色，然而，基于奖励梯度的去噪方法在生成轨迹时需要可微的奖励函数，这限制了其作为通用轨迹优化器的适用性。本文提出Diffusion-ES方法，将无梯度优化与轨迹去噪相结合，以优化黑箱非可微目标，同时保持在数据流形上。Diffusion-ES在进化搜索中从扩散模型中采样轨迹，并使用黑箱奖励函数对其进行评分。通过截断的扩散过程变异高评分轨迹，显著提高了解空间的探索效率。实验表明，Diffusion-ES在nuPlan基准测试中表现优异，超越了现有的采样基础规划器和其他方法。
🔬 方法详解
问题定义：本文旨在解决自主驾驶中的非可微优化问题，现有方法在处理复杂轨迹生成时受限于可微奖励函数的要求。
核心思路：Diffusion-ES通过结合无梯度优化与轨迹去噪，能够在不依赖可微奖励函数的情况下，优化黑箱目标，提升了轨迹生成的灵活性和效率。
技术框架：该方法的整体架构包括从扩散模型中采样轨迹、使用黑箱奖励函数评分、以及通过截断的扩散过程变异高评分轨迹，形成一个高效的进化搜索流程。
关键创新：Diffusion-ES的核心创新在于其无梯度优化能力，使其能够处理非可微的语言形状奖励函数，突破了传统方法的限制。
关键设计：在设计中，Diffusion-ES采用了截断的扩散过程，设置了适当的噪声和去噪步骤，以实现高效的解空间探索。
🖼️ 关键图片
📊 实验亮点
在nuPlan基准测试中，Diffusion-ES展现了卓越的性能，超越了现有的采样基础规划器和其他方法，能够生成如激进变道等复杂行为，解决了传统方法无法应对的最难场景。
🎯 应用场景
该研究具有广泛的应用潜力，尤其在自主驾驶、机器人控制和复杂决策系统中。通过优化非可微目标，Diffusion-ES能够生成更为复杂和灵活的行为，推动智能驾驶技术的进步，提升安全性和效率。
📄 摘要（原文）
Diffusion models excel at modeling complex and multimodal trajectory distributions for decision-making and control. Reward-gradient guided denoising has been recently proposed to generate trajectories that maximize both a differentiable reward function and the likelihood under the data distribution captured by a diffusion model. Reward-gradient guided denoising requires a differentiable reward function fitted to both clean and noised samples, limiting its applicability as a general trajectory optimizer. In this paper, we propose DiffusionES, a method that combines gradient-free optimization with trajectory denoising to optimize black-box non-differentiable objectives while staying in the data manifold. Diffusion-ES samples trajectories during evolutionary search from a diffusion model and scores them using a black-box reward function. It mutates high-scoring trajectories using a truncated diffusion process that applies a small number of noising and denoising steps, allowing for much more efficient exploration of the solution space. We show that DiffusionES achieves state-of-the-art performance on nuPlan, an established closed-loop planning benchmark for autonomous driving. Diffusion-ES outperforms existing sampling-based planners, reactive deterministic or diffusion-based policies, and reward-gradient guidance. Additionally, we show that unlike prior guidance methods, our method can optimize non-differentiable language-shaped reward functions generated by few-shot LLM prompting. When guided by a human teacher that issues instructions to follow, our method can generate novel, highly complex behaviors, such as aggressive lane weaving, which are not present in the training data. This allows us to solve the hardest nuPlan scenarios which are beyond the capabilities of existing trajectory optimization methods and driving policies.