Optimal Policy Sparsification and Low Rank Decomposition for Deep Reinforcement Learning

📄 arXiv: 2403.06313v1 📥 PDF

作者: Vikram Goddla

分类: cs.LG, cs.AI

发布日期: 2024-03-10

💡 一句话要点

提出$L_0$范数正则化以优化深度强化学习策略

🎯 匹配领域: 支柱二：RL算法与架构 (RL & Architecture)

关键词: 深度强化学习 稀疏化 低秩分解 正则化 策略优化 计算资源 过拟合

📋 核心要点

  1. 现有深度强化学习方法训练策略时资源消耗巨大，导致策略过于密集，易于过拟合。
  2. 本文提出了一种基于$L_0$范数的正则化技术，利用最优稀疏图实现DRL策略的稀疏化和低秩分解。
  3. 实验结果显示，$L_0$范数正则化的策略在多个环境中实现了高达93%的稀疏性和显著的性能提升。

📝 摘要（中文）

深度强化学习（DRL）在计算机游戏和机器人等多个领域展现出显著潜力。然而，训练DRL策略消耗大量计算资源，导致密集策略易于过拟合，并限制了其在边缘计算中的实际应用。现有的剪枝和奇异值分解技术虽然可以实现模型压缩，但往往导致性能下降。本文提出了一种新颖的$L_0$范数正则化技术，通过最优稀疏图来稀疏化DRL策略，并促进其降维而不降低奖励。实验结果表明，在多个环境中，$L_0$范数正则化的DRL策略在稀疏性和压缩率上均表现优异，显著优于密集策略。

🔬 方法详解

问题定义：本文旨在解决深度强化学习策略训练过程中资源消耗过大和过拟合的问题。现有的剪枝和奇异值分解技术虽然能实现模型压缩，但往往导致奖励下降，影响策略性能。

核心思路：论文提出了一种新颖的$L_0$范数正则化技术，利用最优稀疏图来稀疏化DRL策略，并促进其降维，旨在在不降低奖励的情况下实现高效的策略压缩。

技术框架：整体架构包括策略稀疏化、低秩分解和性能评估三个主要模块。首先，通过$L_0$范数正则化实现策略的稀疏化，然后进行低秩分解，最后在多个环境中评估策略的性能和稀疏性。

关键创新：最重要的技术创新在于引入$L_0$范数正则化，区别于传统的$L_1$和$L_2$正则化方法，能够在保持策略性能的同时实现更高的稀疏性和压缩率。

关键设计：在实现过程中，关键参数包括稀疏度控制参数和正则化强度，损失函数设计为结合奖励和稀疏性损失，以确保策略在稀疏化过程中仍能保持良好的性能。具体的网络结构设计则依赖于所选用的DRL算法。

📊 实验亮点

实验结果显示，在SuperMarioBros环境中，$L_0$范数正则化的DRL策略实现了93%的稀疏性和70%的压缩率，显著优于密集策略。在Surgical Robot Learning环境中，策略实现了36%的稀疏化和46%的压缩，且性能依然优越，表明该方法的有效性。

🎯 应用场景

该研究的潜在应用领域包括机器人控制、自动驾驶、智能游戏等，能够有效降低计算资源消耗，提升模型在边缘计算环境中的应用能力。未来，随着技术的进一步发展，$L_0$范数正则化可能在更多实际场景中得到应用，推动深度强化学习的普及与发展。

📄 摘要（原文）

Deep reinforcement learning(DRL) has shown significant promise in a wide range of applications including computer games and robotics. Yet, training DRL policies consume extraordinary computing resources resulting in dense policies which are prone to overfitting. Moreover, inference with dense DRL policies limit their practical applications, especially in edge computing. Techniques such as pruning and singular value decomposition have been used with deep learning models to achieve sparsification and model compression to limit overfitting and reduce memory consumption. However, these techniques resulted in sub-optimal performance with notable decay in rewards. $L_1$ and $L_2$ regularization techniques have been proposed for neural network sparsification and sparse auto-encoder development, but their implementation in DRL environments has not been apparent. We propose a novel $L_0$-norm-regularization technique using an optimal sparsity map to sparsify DRL policies and promote their decomposition to a lower rank without decay in rewards. We evaluated our $L_0$-norm-regularization technique across five different environments (Cartpole-v1, Acrobat-v1, LunarLander-v2, SuperMarioBros-7.1.v0 and Surgical Robot Learning) using several on-policy and off-policy algorithms. We demonstrated that the $L_0$-norm-regularized DRL policy in the SuperMarioBros environment achieved 93% sparsity and gained 70% compression when subjected to low-rank decomposition, while significantly outperforming the dense policy. Additionally, the $L_0$-norm-regularized DRL policy in the Surgical Robot Learning environment achieved a 36% sparsification and gained 46% compression when decomposed to a lower rank, while being performant. The results suggest that our custom $L_0$-norm-regularization technique for sparsification of DRL policies is a promising avenue to reduce computational resources and limit overfitting.