Evaluating Collaborative Autonomy in Opposed Environments using Maritime Capture-the-Flag Competitions

📄 arXiv: 2404.17038v1 📥 PDF

作者: Jordan Beason, Michael Novitzky, John Kliem, Tyler Errico, Zachary Serlin, Kevin Becker, Tyler Paine, Michael Benjamin, Prithviraj Dasgupta, Peter Crowley, Charles O'Donnell, John James

分类: cs.RO

发布日期: 2024-04-25

备注: Accepted to the IEEE ICRA Workshop on Field Robotics 2024

💡 一句话要点

评估对抗环境中无人水面车辆的协作自主性

🎯 匹配领域: 支柱一：机器人控制 (Robot Control) 支柱二：RL算法与架构 (RL & Architecture)

关键词: 无人水面车辆 多智能体系统 深度强化学习 行为基础优化 对抗环境 协作算法 Capture-the-Flag 实验评估

📋 核心要点

  1. 现有的多智能体系统在对抗环境中的协作能力不足，尤其是在复杂的动态场景中表现不佳。
  2. 本研究提出了一种结合行为基础优化和深度强化学习的协作算法，以提高无人水面车辆的自主性和竞争力。
  3. 实验结果显示，基于规则的合作方法在CTF竞赛中表现优于深度强化学习方法，未来研究将进一步探索奖励塑造等技术。

📝 摘要（中文）

本研究旨在评估多智能体人工智能方法在对抗环境中部署无人水面车辆（USV）团队的表现。通过Aquaticus测试平台进行的Capture-the-Flag（CTF）风格竞赛，评估了自主代理在真实场景中的表现。采用行为基础优化和深度强化学习（RL）的协作算法，在2023年秋季的竞赛中进行测试。实验结果表明，基于规则的合作方法在行为基础代理中优于深度强化学习训练的代理。未来的研究将进一步整合Pyquaticus环境与MOOS-IvP，以提升CTF竞赛的竞争性，并探讨奖励塑造和模拟到现实的方法。

🔬 方法详解

问题定义：本研究解决的是在对抗环境中多智能体系统的协作自主性不足的问题。现有方法在复杂动态场景下的表现不理想，难以实现有效的团队合作。

核心思路：论文提出了一种结合行为基础优化与深度强化学习的协作算法，通过在真实场景中进行CTF竞赛来评估其有效性。这种设计旨在提升无人水面车辆（USV）在对抗环境中的表现。

技术框架：整体架构包括Aquaticus测试平台和Pyquaticus模拟环境。USV系统被分为两队进行CTF竞赛，算法通过行为基础优化和深度强化学习进行训练和评估。

关键创新：最重要的技术创新在于基于规则的合作方法在行为基础代理中表现优于深度强化学习方法，这一发现挑战了深度学习在多智能体系统中的主导地位。

关键设计：在实验中，采用了特定的参数设置和损失函数，以优化代理的协作能力。网络结构设计上，结合了行为基础优化与深度强化学习的优势，确保了代理在动态环境中的适应性。

🖼️ 关键图片

fig_0
fig_1
fig_2

📊 实验亮点

实验结果表明，基于规则的合作方法在CTF竞赛中显著优于深度强化学习方法，具体表现为在多次对抗中获得了更高的胜率和更好的团队协作效果。这一发现为未来的多智能体系统研究提供了新的视角。

🎯 应用场景

该研究的潜在应用领域包括无人驾驶船只、海洋监测、环境保护等。通过提升无人水面车辆的协作能力，可以在复杂的海洋环境中实现更高效的任务执行，具有重要的实际价值和未来影响。

📄 摘要（原文）

The objective of this work is to evaluate multi-agent artificial intelligence methods when deployed on teams of unmanned surface vehicles (USV) in an adversarial environment. Autonomous agents were evaluated in real-world scenarios using the Aquaticus test-bed, which is a Capture-the-Flag (CTF) style competition involving teams of USV systems. Cooperative teaming algorithms of various foundations in behavior-based optimization and deep reinforcement learning (RL) were deployed on these USV systems in two versus two teams and tested against each other during a competition period in the fall of 2023. Deep reinforcement learning applied to USV agents was achieved via the Pyquaticus test bed, a lightweight gymnasium environment that allows simulated CTF training in a low-level environment. The results of the experiment demonstrate that rule-based cooperation for behavior-based agents outperformed those trained in Deep-reinforcement learning paradigms as implemented in these competitions. Further integration of the Pyquaticus gymnasium environment for RL with MOOS-IvP in terms of configuration and control schema will allow for more competitive CTF games in future studies. As the development of experimental deep RL methods continues, the authors expect that the competitive gap between behavior-based autonomy and deep RL will be reduced. As such, this report outlines the overall competition, methods, and results with an emphasis on future works such as reward shaping and sim-to-real methodologies and extending rule-based cooperation among agents to react to safety and security events in accordance with human experts intent/rules for executing safety and security processes.