Adversarial Imitation Learning via Boosting

📄 arXiv: 2404.08513v1 📥 PDF

作者: Jonathan D. Chang, Dhruv Sreenivas, Yingbing Huang, Kianté Brantley, Wen Sun

分类: cs.LG, cs.AI

发布日期: 2024-04-12

备注: 19 pages, 7 figures, 4 tables, 3 algorithms, ICLR 2024

💡 一句话要点

提出AILBoost以解决现有对抗模仿学习的不足问题

🎯 匹配领域: 支柱二：RL算法与架构 (RL & Architecture)

关键词: 对抗模仿学习 提升算法 弱学习者 重放缓冲区 样本效率 机器人控制 自动驾驶

📋 核心要点

  1. 现有的对抗模仿学习方法如DAC在离线训练中存在不足，无法保证成功的模仿效果。
  2. 本文提出AILBoost算法，通过提升框架维护加权弱学习者集成，训练鉴别器以最大化集成与专家策略的差异。
  3. AILBoost在DeepMind控制套件的状态基础和像素基础环境中均优于DAC，展示了加权重放缓冲区的优势。

📝 摘要（中文）

对抗模仿学习（AIL）在各种模仿学习应用中表现突出，尤其是DAC算法在提高样本效率和可扩展性方面的成功。然而，DAC的原始目标是基于策略的，且其对离线训练的随意应用并不能保证成功的模仿。为了解决这一问题，本文提出了一种新的AIL算法AILBoost，基于提升框架，维护一个加权的弱学习者集成，并训练一个鉴别器以观察集成与专家策略之间的最大差异。通过加权重放缓冲区，AILBoost能够有效利用整个数据集进行训练，实验证明其在多种环境中均优于DAC。

🔬 方法详解

问题定义：本文旨在解决现有对抗模仿学习（AIL）方法在离线训练中的不足，尤其是DAC在应用离线训练时的效果不稳定性。

核心思路：AILBoost算法通过提升框架设计，维护一个加权的弱学习者集成，训练鉴别器以最大化集成策略与专家策略之间的分布差异，从而提高模仿学习的效果。

技术框架：AILBoost的整体架构包括加权重放缓冲区、弱学习者集成和鉴别器训练三个主要模块。加权重放缓冲区用于存储和管理状态-动作分布，弱学习者集成则通过加权策略进行训练。

关键创新：AILBoost的核心创新在于利用提升框架对弱学习者进行加权，从而有效利用历史数据，解决了传统方法在离线训练中的数据利用不足问题。

关键设计：在算法实现中，重放缓冲区的数据贡献根据提升框架的权重进行折扣，确保较旧策略的数据对训练的影响得到合理控制。

🖼️ 关键图片

fig_0
fig_1
fig_2

📊 实验亮点

AILBoost在DeepMind控制套件的实验中表现优异，尤其在状态基础环境中超越DAC和ValueDICE，显示出其在样本效率和训练效果上的显著提升，尤其在仅使用一条专家轨迹的情况下，依然能达到竞争性能。

🎯 应用场景

AILBoost算法具有广泛的应用潜力，特别是在机器人控制、自动驾驶和游戏AI等领域。通过提高模仿学习的效率和效果，该算法能够帮助系统更好地学习复杂的任务，提升自主决策能力，具有重要的实际价值和未来影响。

📄 摘要（原文）

Adversarial imitation learning (AIL) has stood out as a dominant framework across various imitation learning (IL) applications, with Discriminator Actor Critic (DAC) (Kostrikov et al.,, 2019) demonstrating the effectiveness of off-policy learning algorithms in improving sample efficiency and scalability to higher-dimensional observations. Despite DAC's empirical success, the original AIL objective is on-policy and DAC's ad-hoc application of off-policy training does not guarantee successful imitation (Kostrikov et al., 2019; 2020). Follow-up work such as ValueDICE (Kostrikov et al., 2020) tackles this issue by deriving a fully off-policy AIL objective. Instead in this work, we develop a novel and principled AIL algorithm via the framework of boosting. Like boosting, our new algorithm, AILBoost, maintains an ensemble of properly weighted weak learners (i.e., policies) and trains a discriminator that witnesses the maximum discrepancy between the distributions of the ensemble and the expert policy. We maintain a weighted replay buffer to represent the state-action distribution induced by the ensemble, allowing us to train discriminators using the entire data collected so far. In the weighted replay buffer, the contribution of the data from older policies are properly discounted with the weight computed based on the boosting framework. Empirically, we evaluate our algorithm on both controller state-based and pixel-based environments from the DeepMind Control Suite. AILBoost outperforms DAC on both types of environments, demonstrating the benefit of properly weighting replay buffer data for off-policy training. On state-based environments, DAC outperforms ValueDICE and IQ-Learn (Gary et al., 2021), achieving competitive performance with as little as one expert trajectory.