Multi-Agent Reinforcement Learning for Maritime Operational Technology Cyber Security
作者: Alec Wilson, Ryan Menzies, Neela Morarji, David Foster, Marco Casassa Mont, Esin Turkbeyler, Lisa Gralewski
分类: cs.LG, cs.CR, cs.MA
发布日期: 2024-01-18
备注: 13 pages, 7 figures, Proceedings of the Conference on Applied Machine Learning in Information Security 2023 (CAMLIS)
期刊: Proceedings of the Conference on Applied Machine Learning in Information Security 2023 (CAMLIS), Arlington VA, USA, October 19-20, 2023, CEUR-WS.org, online CEUR-WS.org/Vol-3652/paper3.pdf
💡 一句话要点
提出多智能体强化学习以解决海洋操作技术网络安全问题
🎯 匹配领域: 支柱二：RL算法与架构 (RL & Architecture)
关键词: 多智能体强化学习 网络安全 工业控制系统 自主防御 海洋操作技术 模拟环境 策略优化
📋 核心要点
- 现有的OT网络防御措施相对不成熟，主要由于基础设施脆弱和传统IT防御方案的局限性。
- 本文提出了基于多智能体强化学习的自主网络防御决策框架，利用IPMSRL模拟环境进行研究。
- 实验结果表明，MAPPO在800K时间步后达到了最优策略，而IPPO在一百万时间步后仅达到了0.966的平均结果。
📝 摘要（中文）
本文展示了自主网络防御在工业控制系统中的应用潜力，并提供了一个基线环境以进一步探索多智能体强化学习（MARL）在该领域的应用。研究引入了一个通用的集成平台管理系统（IPMS）模拟环境IPMSRL，并探讨了MARL在海洋基础的IPMS操作技术中的自主网络防御决策。由于OT基础设施的脆弱性及传统IT网络防御解决方案的局限性，OT网络防御行动相对不成熟。实验中，采用共享评论者实现的多智能体近端策略优化（MAPPO）在性能上优于独立近端策略优化（IPPO），并在800K时间步后达到了最优策略。
🔬 方法详解
问题定义：本文旨在解决海洋操作技术（OT）中的网络安全防御问题，现有方法面临基础设施脆弱和传统IT防御措施不足的挑战。
核心思路：通过引入多智能体强化学习（MARL），实现自主网络防御决策，提升OT环境下的安全性。设计上采用共享评论者机制以提高学习效率。
技术框架：整体架构包括IPMSRL模拟环境、MARL算法（MAPPO和IPPO），以及攻击检测机制。主要模块包括环境建模、策略优化和决策执行。
关键创新：最重要的创新在于采用共享评论者的MAPPO算法，相较于传统的IPPO，显著提升了策略学习的效率和效果。
关键设计：在超参数调优方面，经过调整的超参数显著提升了训练性能，达到了最优策略，而默认超参数仅偶尔获胜，且大多数模拟结果为平局。
🖼️ 关键图片
📊 实验亮点
实验结果显示，MAPPO在800K时间步后达到了最优策略，平均结果为1，而IPPO在一百万时间步后仅达到了0.966。此外，当攻击检测警报成功率降低至0.75或0.9时，MARL防御者仍能在超过97.5%或99.5%的回合中获胜，显示出其强大的鲁棒性。
🎯 应用场景
该研究的潜在应用领域包括海洋操作技术的网络安全防御、工业控制系统的自主防护以及智能化网络安全解决方案的开发。通过提升OT环境的安全性，能够有效应对日益复杂的网络攻击，具有重要的实际价值和未来影响。
📄 摘要（原文）
This paper demonstrates the potential for autonomous cyber defence to be applied on industrial control systems and provides a baseline environment to further explore Multi-Agent Reinforcement Learning's (MARL) application to this problem domain. It introduces a simulation environment, IPMSRL, of a generic Integrated Platform Management System (IPMS) and explores the use of MARL for autonomous cyber defence decision-making on generic maritime based IPMS Operational Technology (OT). OT cyber defensive actions are less mature than they are for Enterprise IT. This is due to the relatively brittle nature of OT infrastructure originating from the use of legacy systems, design-time engineering assumptions, and lack of full-scale modern security controls. There are many obstacles to be tackled across the cyber landscape due to continually increasing cyber-attack sophistication and the limitations of traditional IT-centric cyber defence solutions. Traditional IT controls are rarely deployed on OT infrastructure, and where they are, some threats aren't fully addressed. In our experiments, a shared critic implementation of Multi Agent Proximal Policy Optimisation (MAPPO) outperformed Independent Proximal Policy Optimisation (IPPO). MAPPO reached an optimal policy (episode outcome mean of 1) after 800K timesteps, whereas IPPO was only able to reach an episode outcome mean of 0.966 after one million timesteps. Hyperparameter tuning greatly improved training performance. Across one million timesteps the tuned hyperparameters reached an optimal policy whereas the default hyperparameters only managed to win sporadically, with most simulations resulting in a draw. We tested a real-world constraint, attack detection alert success, and found that when alert success probability is reduced to 0.75 or 0.9, the MARL defenders were still able to win in over 97.5% or 99.5% of episodes, respectively.