Dissecting Deep RL with High Update Ratios: Combatting Value Divergence
作者: Marcel Hussing, Claas Voelcker, Igor Gilitschenski, Amir-massoud Farahmand, Eric Eaton
分类: cs.LG, cs.AI
发布日期: 2024-03-09 (更新: 2024-08-05)
备注: Accepted as a conference paper at the First Reinforcement Learning Conference (RLC)
💡 一句话要点
提出高更新比下的深度强化学习以解决价值函数发散问题
🎯 匹配领域: 支柱二：RL算法与架构 (RL & Architecture)
📋 核心要点
- 现有深度强化学习方法在大更新比下容易出现价值函数发散，导致学习能力下降。
- 本文提出单位球归一化方法，旨在解决价值函数发散问题，从而提高学习效果。
- 在dm_control套件上进行实验，结果显示该方法在狗类任务上表现优异，超越了传统模型。
- method_zh
📝 摘要（中文）
我们展示了深度强化学习算法在梯度更新次数远超环境样本数量的情况下，仍能保持学习能力，而无需重置网络参数，进而对抗价值函数发散。在大更新与数据比下，Nikishin等（2022）提出的初始偏差现象使得智能体过拟合早期交互，忽视后期经验，影响学习能力。本文探讨导致初始偏差的现象，发现价值函数发散是一个根本挑战。我们提出了一种简单的单位球归一化方法，使得在大更新比下仍能有效学习，并在广泛使用的dm_control套件上展示了其有效性，在具有挑战性的狗类任务上取得了与基于模型的方法相竞争的表现。我们的结果部分质疑了早期数据过拟合导致次优学习的先前解释。
🖼️ 关键图片
📄 摘要（原文）
We show that deep reinforcement learning algorithms can retain their ability to learn without resetting network parameters in settings where the number of gradient updates greatly exceeds the number of environment samples by combatting value function divergence. Under large update-to-data ratios, a recent study by Nikishin et al. (2022) suggested the emergence of a primacy bias, in which agents overfit early interactions and downplay later experience, impairing their ability to learn. In this work, we investigate the phenomena leading to the primacy bias. We inspect the early stages of training that were conjectured to cause the failure to learn and find that one fundamental challenge is a long-standing acquaintance: value function divergence. Overinflated Q-values are found not only on out-of-distribution but also in-distribution data and can be linked to overestimation on unseen action prediction propelled by optimizer momentum. We employ a simple unit-ball normalization that enables learning under large update ratios, show its efficacy on the widely used dm_control suite, and obtain strong performance on the challenging dog tasks, competitive with model-based approaches. Our results question, in parts, the prior explanation for sub-optimal learning due to overfitting early data.