Multi-Agent Reinforcement Learning of Co-Adaptive Hip Exoskeleton Control (MARL-Exo) for Metabolic Energy Reduction
作者: Yifei Yuan, Jakob Wolf, Ghaith Androwis, Xianlian Zhou
分类: cs.RO, cs.LG
发布日期: 2026-10-27
备注: 14 pages, 6 figures, including supplementary material
💡 一句话要点
提出MARL-Exo以降低步态行走的代谢能耗问题
🎯 匹配领域: 支柱二：RL算法与架构 (RL & Architecture)
关键词: 多智能体强化学习 外骨骼控制 代谢能耗 步态分析 康复技术 生理模型 机械功率塑形
📋 核心要点
- 现有方法在个体化调优上存在挑战，难以实现普适性和高效性。
- 提出MARL-Exo框架，通过多智能体强化学习实现外骨骼与人类模型的共同适应，优化代谢能耗。
- 实验结果表明，Metabolic策略在模拟中减少了13.8%的代谢功率，实际测试中减少了18.3%的净代谢率。
📝 摘要（中文）
本研究旨在探讨在代谢能量奖励下，是否可以通过与肌肉骨骼人模型共同适应的模拟训练的髋部外骨骼策略，减少新用户的步态代谢成本，而无需针对每个用户进行调优。同时，研究还考察了机械功率塑形对学习到的辅助效果的影响。我们开发了MARL-Exo，一个多智能体强化学习框架，其中全身人模型与双侧髋部外骨骼策略在两个阶段中并行训练。结果显示，Metabolic策略在模拟中相较于无设备行走减少了13.8%的肌肉代谢功率，而在硬件测试中，MARL-Exo在所有参与者中相较于被动状态减少了18.3%的净代谢率。
🔬 方法详解
问题定义：本研究旨在解决现有外骨骼控制策略在新用户中应用时的代谢能耗问题。现有方法通常需要针对每个用户进行个性化调优，限制了其普适性和应用范围。
核心思路：论文提出的MARL-Exo框架通过多智能体强化学习，使得外骨骼与肌肉骨骼模型在代谢能量奖励下共同适应，从而实现无需个体调优的代谢成本降低。
技术框架：MARL-Exo框架分为两个阶段，首先在模拟环境中训练全身人模型与外骨骼策略，然后将训练好的策略应用于真实用户。外骨骼仅观察延迟的髋部运动学和过去的命令，其奖励包括基于生理的能量模型计算的肌肉代谢功率。
关键创新：该研究的创新点在于提出了一种基于群体水平的控制策略，避免了传统方法中每个用户的个性化调优，同时在模拟中筛选出有效的辅助目标。
关键设计：在损失函数设计上，采用了代谢功率作为奖励信号，结合机械功率塑形进行训练。模型中包含304个肌肉驱动器，具有随机解剖结构、步态和速度，以增强模型的泛化能力。
🖼️ 关键图片
📊 实验亮点
实验结果显示，Metabolic策略在模拟中减少了13.8%的肌肉代谢功率，而在实际测试中，MARL-Exo在所有参与者中相较于被动状态减少了18.3%的净代谢率。相较于无外骨骼行走，平均减少了7.4%的代谢成本，六名参与者的代谢成本降低幅度超过了设备佩戴的惩罚。
🎯 应用场景
该研究的潜在应用领域包括康复医学、老年人辅助行走设备以及运动训练等。通过降低步态的代谢能耗，MARL-Exo能够为用户提供更高效的行走辅助，改善生活质量，并在未来可能推动外骨骼技术的广泛应用。
📄 摘要（原文）
Objective: This study aims to determine whether a hip-exoskeleton policy trained in simulation, by co-adapting with a musculoskeletal human model under a metabolic-energy reward, reduces the metabolic cost of walking in new users without per-user tuning, and how mechanical-power shaping affects the learned assistance. Methods: We developed MARL-Exo, a multi-agent reinforcement learning framework in which a full-body human model (304 muscle actuators; randomized anatomy, gait, and speed) and a bilateral hip-exoskeleton policy are trained concurrently in two stages. The exoskeleton observes only delayed hip kinematics and its past commands; its reward includes muscle metabolic power from a physiologically based energetics model, alone (Metabolic) or with mechanical-power shaping (Metabolic + Power). The Metabolic policy was deployed and tested without retraining in eight healthy adults walking at 1.25 m/s, using indirect calorimetry. Results: In simulation, the Metabolic policy reduced modeled muscle metabolic power by 13.8% relative to no-device walking, versus 8.8% for Metabolic + Power. On hardware, MARL-Exo reduced net metabolic rate by 18.3 $\pm$ 6.6% relative to Passive in all eight participants. Relative to No-Exo, the mean reduction was 7.4 $\pm$ 10.0%; six participants more than offset the 13.1% device-wearing penalty, walking up to 24.2% below their No-Exo cost. Conclusion: A metabolically informed policy co-adapted in simulation transferred to users without per-user tuning and consistently reduced the metabolic cost of walking with the device, while explicit power shaping reduced the simulated benefit. Significance: Unlike per-user human-in-the-loop optimization, this approach yields a single population-level controller and allows assistance objectives to be screened in simulation before human testing.