Risk-Sensitive Mission Planning for Multi-Target On-Orbit Inspection via Distributional Reinforcement Learning

📄 arXiv: 2610.41647v1 📥 PDF

作者: Adam Abdin

分类: cs.AI, math.OC

发布日期: 2026-10-30

期刊: 77th International Astronautical Congress (IAC), International Astronautical Federation (IAF), Oct 2026, Antalya (Turkey), Turkey

💡 一句话要点

提出基于分布式强化学习的风险敏感多目标轨道检查任务规划方法

🎯 匹配领域: 支柱二：RL算法与架构 (RL & Architecture)

关键词: 轨道检查 风险敏感规划 分布式强化学习 量子回归深度Q网络 多目标任务 航天任务 不确定性决策

📋 核心要点

  1. 现有的多目标轨道服务任务规划方法未能有效应对动态变化的操作环境和结果的不确定性，导致任务成功率降低。
  2. 本文提出了一种基于分布式强化学习的适应性规划方法，通过量子回归深度Q网络实现风险敏感决策，考虑任务回报的期望和方差。
  3. 实验结果显示，风险厌恶策略能够将任务回报的方差减少约一半，并降低任务损失的概率，尽管平均回报有所下降。

📝 摘要（中文）

多目标轨道服务任务需要能够适应不断变化的操作条件的规划工具，同时考虑到安全关键的航天飞行中固有的结果变异性。本文提出了一种适应性规划方法，应用于低地球轨道的多目标服务任务，特别是低推力服务器的检查任务。该方法基于分布式强化学习，学习任务回报的完整分布，从而支持每个规划步骤的明确风险敏感决策。具体而言，量子回归深度Q网络（QR-DQN）代理通过均值-方差标准共同考虑预测回报的期望和方差，从而选择下一个检查目标。实验结果表明，风险厌恶代理相较于风险中性代理显著降低了任务回报的方差，并降低了任务损失的概率。

🔬 方法详解

问题定义：本文旨在解决多目标轨道检查任务中，如何在动态环境下进行有效的风险敏感规划。现有方法往往忽视了结果的不确定性，导致任务成功率降低。

核心思路：论文提出了一种基于分布式强化学习的适应性规划方法，利用量子回归深度Q网络（QR-DQN）来同时考虑任务回报的期望和方差，从而实现风险敏感的决策。

技术框架：整体架构包括环境建模、状态观测、决策制定和策略优化四个主要模块。环境模型考虑了轨道元素、访问历史、目标风险状态和剩余任务时间预算。

关键创新：最重要的技术创新在于引入了风险敏感的决策机制，通过学习任务回报的完整分布，显著提高了在不确定环境下的决策质量。与传统的风险中性方法相比，能够更好地应对任务失败的风险。

关键设计：关键参数包括风险厌恶权重的设置，损失函数设计为均值-方差标准，网络结构采用深度Q网络架构，能够有效捕捉任务回报的分布特征。通过数值实验验证了所提方法的有效性。

📊 实验亮点

实验结果表明，风险厌恶代理在相同环境和奖励条件下，能够将任务回报的方差减少约50%，并降低任务损失的概率。与风险中性代理相比，风险厌恶策略在均值回报上接近高效前沿，显示出显著的性能提升。

🎯 应用场景

该研究的潜在应用领域包括航天任务规划、无人机巡检和机器人自主导航等。通过提高任务规划的风险敏感性，可以显著提升在复杂和不确定环境下的任务成功率，具有重要的实际价值和未来影响。

📄 摘要（原文）

Multi-target on-orbit servicing missions require planning tools that can adapt to evolving operational conditions while accounting for the outcome variability inherent to safety-critical spaceflight. This paper presents an adaptive planning approach for multi-target servicing campaigns in low Earth orbit and applies it to inspection missions flown by a low-thrust servicer. The approach is based on Distributional Reinforcement Learning, which learns the full distribution of mission returns and thereby supports explicit risk-sensitive decision-making at each planning step. Specifically, a Quantile-Regression Deep Q-Network (QR-DQN) agent selects which target to inspect next by jointly considering the expectation and the variance of the predicted return through a mean-variance criterion. The mission environment models circular orbits with a spread in inclination and uses Edelbaum's combined-manoeuvre low-thrust transfer estimates. The observation comprises the orbital elements, the visit history, the observed risk state of each target and the remaining mission-time budget. The risk state of each target changes stochastically during the campaign, making the outcome of an inspection uncertain and creating operationally meaningful distinctions between risk-neutral and risk-averse planning strategies. Because the mission-time budget generally admits only a subset of the candidate targets, the planner must also select which targets to inspect. Numerical experiments on scenarios with five target satellites show that one risk-aversion weight moves the policy across a range of mean return and return variance, over an unchanged environment and reward. Risk-averse agents reduce the variance of the episodic return by approximately one half relative to risk-neutral agents and lower the probability of a mission loss, at a cost in mean return. To validate the learned policies, the efficient frontier, which gives the highest mean return attainable at each level of return variance, is computed exactly by enumerating the admissible visit orders. At equal variance, the learned policies lie close to this frontier in mean return, and the risk-averse policies have a lower probability of a mission loss than the frontier policies.