Centralized vs. Decentralized Multi-Agent Reinforcement Learning for Enhanced Control of Electric Vehicle Charging Networks

📄 arXiv: 2404.12520v1 📥 PDF

作者: Amin Shojaeighadikolaei, Zsolt Talata, Morteza Hashemi

分类: cs.AI

发布日期: 2024-04-18

备注: 12 pages, 9 figures

💡 一句话要点

提出CTDE-DDPG以优化电动车充电网络控制

🎯 匹配领域: 支柱二：RL算法与架构 (RL & Architecture)

关键词: 电动车充电 多智能体强化学习 集中训练 分散执行 深度学习 智能电网 需求侧管理

📋 核心要点

  1. 现有电动车充电控制方法主要基于集中模型，难以应对电力需求波动和环境不确定性。
  2. 本文提出CTDE-DDPG方法，通过集中训练和分散执行的方式，实现EV之间的合作与优化充电策略。
  3. 实验结果显示，CTDE-DDPG在充电效率和成本方面均有显著提升，降低了充电成本和总变动。

📝 摘要（中文）

随着电动车（EV）的广泛应用，电力分配网络和智能电网基础设施面临着显著的电力需求挑战，尤其是在高峰时段。本文提出了一种基于多智能体强化学习（MARL）框架的分布式充电策略，利用深度确定性策略梯度（DDPG）算法，针对住宅社区中的EV进行优化充电控制。该方法采用集中训练、分散执行（CTDE）策略，在训练阶段促进智能体之间的合作，同时在执行阶段保持分散和隐私保护。理论和数值结果表明，CTDE-DDPG框架显著提高了充电效率，平均降低了总变动约36%和充电成本约9.1%。

🔬 方法详解

问题定义：本文旨在解决电动车充电控制中的集中模型带来的效率低下和环境不确定性问题。现有方法在应对电力需求波动时表现不佳，难以实现最优充电策略。

核心思路：提出CTDE-DDPG方法，通过集中训练和分散执行的方式，利用多智能体强化学习框架来优化电动车的充电策略，确保在执行阶段的隐私保护和分散性。

技术框架：该方法包括集中训练阶段，所有智能体共享信息以提高学习效率；执行阶段则由各个智能体独立运行，确保系统的灵活性和隐私。主要模块包括环境建模、智能体训练和执行策略。

关键创新：CTDE-DDPG框架的核心创新在于结合了集中训练与分散执行的优势，克服了传统集中模型的局限性，提升了充电效率和系统的可扩展性。

关键设计：在参数设置上，采用了适应性学习率和经验回放机制，损失函数设计为结合了策略梯度和价值函数的复合形式，以提高训练的稳定性和收敛速度。

🖼️ 关键图片

fig_0
fig_1
fig_2

📊 实验亮点

实验结果表明，CTDE-DDPG框架在充电效率上提高了约36%，充电成本平均降低了9.1%。这些结果显示出该方法在应对电力需求波动和优化充电策略方面的显著优势，具有良好的应用前景。

🎯 应用场景

该研究的潜在应用领域包括智能电网管理、电动车充电站的优化调度以及需求侧管理。通过优化充电策略，可以有效降低电力成本，提高电网的稳定性，促进可再生能源的利用，具有重要的实际价值和社会影响。

📄 摘要（原文）

The widespread adoption of electric vehicles (EVs) poses several challenges to power distribution networks and smart grid infrastructure due to the possibility of significantly increasing electricity demands, especially during peak hours. Furthermore, when EVs participate in demand-side management programs, charging expenses can be reduced by using optimal charging control policies that fully utilize real-time pricing schemes. However, devising optimal charging methods and control strategies for EVs is challenging due to various stochastic and uncertain environmental factors. Currently, most EV charging controllers operate based on a centralized model. In this paper, we introduce a novel approach for distributed and cooperative charging strategy using a Multi-Agent Reinforcement Learning (MARL) framework. Our method is built upon the Deep Deterministic Policy Gradient (DDPG) algorithm for a group of EVs in a residential community, where all EVs are connected to a shared transformer. This method, referred to as CTDE-DDPG, adopts a Centralized Training Decentralized Execution (CTDE) approach to establish cooperation between agents during the training phase, while ensuring a distributed and privacy-preserving operation during execution. We theoretically examine the performance of centralized and decentralized critics for the DDPG-based MARL implementation and demonstrate their trade-offs. Furthermore, we numerically explore the efficiency, scalability, and performance of centralized and decentralized critics. Our theoretical and numerical results indicate that, despite higher policy gradient variances and training complexity, the CTDE-DDPG framework significantly improves charging efficiency by reducing total variation by approximately %36 and charging cost by around %9.1 on average...