MemOPD: On-Policy Distillation through Memory State Alignment for Long-Horizon Agents

📄 arXiv: 2608.07068v1 📥 PDF

作者: Zhiyuan Liu, Tinghong Ye, Chenghao Liu, Yizhuo Li, Songfang Huang

分类: cs.AI

发布日期: 2026-08-07

🔗 代码/项目: GITHUB

💡 一句话要点

提出MemOPD以解决长时间代理的记忆状态对齐问题

🎯 匹配领域: 支柱二：RL算法与架构 (RL & Architecture)

关键词: 长时间代理 记忆对齐 在线蒸馏 稀疏奖励 性能提升

📋 核心要点

  1. 现有方法在长时间代理的记忆管理中面临稀疏奖励指导不足的问题，导致性能下降。
  2. 本文提出MemOPD，通过记忆状态对齐实现在线蒸馏，确保教师评分与学生的状态一致。
  3. 实验结果显示，MemOPD在F1分数上相较于传统方法有显著提升，并加快了训练速度。

📝 摘要（中文）

长时间代理在交互过程中积累的上下文会影响其性能和稳定性。紧凑的记忆通过压缩和重写历史来缓解这一问题，但现有方法依赖于最终任务奖励的近端策略优化（PPO），在稀疏奖励下对单个记忆更新指导不足。为此，本文提出了基于记忆对齐的在线蒸馏（MemOPD），通过记录模型调用的输入和输出，恢复原始的令牌位置和因果可见性，从而实现有效的教师评分。实验表明，MemOPD在多个上下文更新中验证了状态对齐，并在匹配控制下提高了F1分数7.0%。MemOPD-3B在F1分数上相较于PPO提升高达416.2%，并在训练期间实现了高达1.63倍的计算速度提升。

🔬 方法详解

问题定义：本文旨在解决长时间代理在交互过程中因上下文积累导致的性能和稳定性问题。现有方法依赖于稀疏奖励进行记忆更新，缺乏有效指导，导致状态对齐失效。

核心思路：MemOPD通过记录每次模型调用的输入和输出，恢复原始的令牌位置和因果可见性，从而实现教师对学生的有效监督，确保在相同状态下进行评分。

技术框架：MemOPD的整体架构包括输入记录、输出重构、状态恢复和教师评分四个主要模块。首先记录模型调用的输入和输出，然后重构这些调用以恢复原始状态，最后进行教师评分。

关键创新：MemOPD的核心创新在于通过记忆状态对齐实现在线蒸馏，确保教师评分与学生的状态一致。这一方法克服了传统方法中因上下文重写导致的状态不一致问题。

关键设计：在MemOPD中，关键设计包括对输入和输出的精确记录、重构过程中的状态恢复，以及教师评分时的全词汇监督。这些设计确保了模型在训练过程中的高效性和准确性。

🖼️ 关键图片

fig_0
fig_1
fig_2

📊 实验亮点

实验结果表明，MemOPD在多个上下文更新中实现了状态对齐，相较于传统的持久历史教师评分，F1分数提升了7.0%。MemOPD-3B在F1分数上较PPO提升高达416.2%，并在训练期间实现了高达1.63倍的计算速度提升，显示出其显著的性能优势。

🎯 应用场景

MemOPD的研究成果在长时间决策任务中具有广泛的应用潜力，如机器人控制、自动驾驶和智能助手等领域。通过提高代理的性能和稳定性，能够在复杂环境中实现更高效的决策支持，推动智能系统的实际应用和发展。

📄 摘要（原文）

Long-horizon agents accumulate growing contexts during interaction, impairing performance and stability. Compact memory mitigates this problem by compressing and rewriting the history retained between model invocations. Learning what to retain typically relies on proximal policy optimization (PPO) with final task rewards, but sparse rewards provide little guidance for individual memory updates. This limitation motivates on-policy distillation (OPD), which supplies dense teacher supervision on student rollouts. For such supervision to be valid, the teacher must evaluate each sampled action under the same state in which it was generated. However, the context rewriting performed during memory compression can break this alignment. When sampled responses are retained and re-encoded for later invocations, flattening the interaction into a persistent history may cause the teacher to score the action under a state that the student never visited during rollout. The action therefore remains on-policy by provenance, but not necessarily by state. We therefore propose Memory-Aligned On-Policy Distillation (MemOPD). MemOPD records the inputs and sampled outputs of each model invocation, restores its original token positions and causal visibility, and packs the reconstructed invocations for efficient teacher scoring. The teacher provides full-vocabulary supervision at the sampled action positions, while PPO preserves the final task objective. Experiments verify state alignment across several context updates and show that it improves F1 by 7.0% over persistent-history teacher scoring in a matched control. Overall, MemOPD-3B improves F1 over PPO by up to 416.2%, while packing yields up to a 1.63x speedup in actor computation during training. The code for this work is publicly available at: https://github.com/TPssp/MemOPD.