Procedural Memory Distillation: Online Reflection for Self-Improving Language Models

📄 arXiv: 2607.01480 📥 PDF

作者: Ye Liu, Srijan Bansal, Bo Pang, Yang Li, Zeyu Leo Liu, Yifei Ming, Zixuan Ke, Shafiq Joty, Semih Yavuz

分类: cs.AI, cs.LG

发布日期: 2026-07-05

💡 一句话要点

提出程序记忆蒸馏以提升自我改进语言模型的性能

🎯 匹配领域: 支柱二：RL算法与架构 (RL & Architecture)

关键词: 程序记忆蒸馏 自我改进 强化学习 策略更新 跨回合学习 自然语言处理 智能助手

📋 核心要点

  1. 现有的强化学习方法在处理跨回合信息时存在不足，无法有效利用回合间的策略变化和经验积累。
  2. 本文提出的程序记忆蒸馏（PMD）通过将跨回合信号转化为可重用的程序记忆，增强了策略的学习能力。
  3. 实验结果显示，PMD在多个基准测试中显著提升了模型性能，验证了其有效性和优越性。

📝 摘要（中文）

强化学习与可验证奖励（RLVR）及其自蒸馏变体（如SDPO）在每次回合中评估策略并更新。然而，回合中的丰富程序信息往往未被保留或重用。本文提出程序记忆蒸馏（PMD），将跨回合信号转化为可重用的程序记忆，并在训练过程中将其蒸馏到策略权重中。PMD通过原始轨迹、自我反思策略和更高层次行为模式三个抽象层次组织记忆，支持模型在推理时无记忆依赖。实验表明，PMD在Qwen3-8B和OLMo3-Instruct-7B上相较于SDPO在SCIKNOWEVAL上提升3.8-5.5%，在LIVECODEBENCH上提升7.9-13.6%。

🔬 方法详解

问题定义：本文旨在解决现有强化学习方法在回合间缺乏有效信息保留和重用的问题，导致策略更新效率低下。

核心思路：PMD通过将跨回合信号转化为程序记忆，形成一个自我监督的学习框架，使得模型能够在训练中吸收和利用这些经验。

技术框架：PMD的整体架构包括三个主要模块：原始轨迹的收集、自我反思策略的提取和高层次行为模式的识别，所有这些都在线提取自模型的轨迹。

关键创新：PMD的核心创新在于其共进化设计，策略生成的回合更新记忆，而记忆又反过来影响策略的监督，形成闭环学习机制。

关键设计：在实现上，PMD采用了记忆条件的自教师机制，利用积累的经验来指导学生模型的学习，确保其逐步内化程序知识。具体的损失函数和网络结构细节在论文中进行了详细描述。

📊 实验亮点

实验结果显示，PMD在Qwen3-8B和OLMo3-Instruct-7B模型上，相较于SDPO在SCIKNOWEVAL上提升了3.8-5.5%，在LIVECODEBENCH上提升了7.9-13.6%。冻结记忆或策略会导致PMD性能下降超过10%，验证了共进化设计的重要性。

🎯 应用场景

该研究的潜在应用领域包括自然语言处理、对话系统和智能助手等，能够显著提升模型在动态环境中的适应能力和学习效率。未来，PMD可能推动自我改进模型在更复杂任务中的应用，提升人工智能系统的智能水平。

📄 摘要（原文）

Reinforcement learning with verifiable rewards (RLVR), along with recent selfdistillation variants such as SDPO, evaluates each rollout against a verifier and updates the policy from that episode-level signal. However, the richer procedural information in the rollout is rarely retained or reused. Across episodes and epochs, the model repeatedly encounters related problems under a changing policy, producing cross-episode signals that episode-local updates cannot capture: which strategies consistently pass verification, which failure modes persist, which patterns recur. We propose Procedural Memory Distillation (PMD), which converts these crossepisode signals into reusable procedural memory and distills it into the policy's weights during training. This memory functions as a training scaffold, absorbed into the policy itself, yielding a memory-free model at inference. PMD organizes the memory at three levels of abstraction: raw trajectories, self-reflected strategies and lessons, and higher-level behavioral patterns that recur across problems, all extracted online from the model's own trajectories. A memory-conditioned self-teacher draws on the accumulated experience to supervise the student on its own rollouts, enabling student to progressively internalize procedural knowledge within its parameters. The central design principle is co-evolution: the policy generates rollouts that update the memory, and memory shapes the supervision that updates the policy. Empirically, across Qwen3-8B and OLMo3-Instruct-7B, PMD improves over SDPO by 3.8-5.5% on SCIKNOWEVAL and 7.9-13.6% on LIVECODEBENCH. Co-evolution powers these gains: freezing either the memory or the policy trails PMD by more than 10% across SCIKNOWEVAL domains.