Learning from Your Own Mistakes: Constructing Learnable Micro-Reflective Trajectories for Self-Distillation
作者: Zhilin Huang, Hang Gao, Ziqiang Dong, Yuan Chen, Yifeng Luo, Chujun Qin, Jingyi Wang, Yang Yang, Guanjun Jiang
分类: cs.LG
发布日期: 2026-06-17
💡 一句话要点
提出TAPO以解决自蒸馏中的错误诊断问题
🎯 匹配领域: 支柱二：RL算法与架构 (RL & Architecture) 支柱九：具身大模型 (Embodied Foundation Models)
关键词: 自蒸馏 轨迹优化 错误修正 自然语言处理 强化学习
📋 核心要点
- 现有自蒸馏方法通过隐式对齐来改善推理，但缺乏对模型错误的具体诊断和纠正指导。
- 本文提出TAPO，通过构建微反射轨迹，利用模型自身的错误输出进行显式的修正和指导。
- 实验结果显示，TAPO在AIME 2024、AIME 2025和HMMT 2025上相较于GRPO取得了显著的性能提升。
📝 摘要（中文）
自蒸馏通过使用模型自身的输出作为训练信号来改善大型语言模型的推理能力，然而现有方法缺乏对模型具体错误的诊断和纠正指导。本文提出了轨迹增强策略优化（TAPO），将自蒸馏从隐式分布对齐提升到显式轨迹构建。TAPO利用模型在同一查询下生成的正确和错误输出之间的对比，构建微反射修正，保留模型错误推理直至失败点，并插入自然语言诊断和基于正确参考的修正推理。实验结果表明，TAPO在相同训练步骤下相较于GRPO取得了一致性提升，增强了首次推理和错误修正的有效性。
🔬 方法详解
问题定义：本文旨在解决自蒸馏过程中模型对自身错误缺乏有效诊断的问题。现有方法主要依赖隐式的分布对齐，无法提供细粒度的错误修正指导。
核心思路：TAPO的核心思路是通过对比模型生成的正确和错误输出，构建微反射修正轨迹，从而实现显式的错误诊断和修正。这样设计的目的是为了让模型在学习中更好地理解自身的错误。
技术框架：TAPO的整体架构包括两个主要阶段：首先，模型在RL训练中生成正确和错误的输出；其次，利用这些输出构建微反射修正轨迹，并插入自然语言的诊断信息。
关键创新：TAPO的主要创新在于将自蒸馏从隐式对齐转变为显式轨迹构建，这一方法使得模型能够更好地保留自身的推理分布，而不是简单模仿一个特权分布。
关键设计：TAPO引入了难度感知的候选选择机制，以确保在模型能力边界内进行有效学习，同时采用解耦的优势估计来防止梯度污染。
🖼️ 关键图片
📊 实验亮点
实验结果表明，TAPO在AIME 2024、AIME 2025和HMMT 2025上相较于GRPO在相同训练步骤下取得了显著提升，增强了模型的首次推理和错误修正的有效性，具体性能数据未提供。
🎯 应用场景
该研究的潜在应用领域包括自然语言处理、智能问答系统和对话系统等。通过改进自蒸馏技术，TAPO能够提升模型的推理能力和错误修正能力，具有重要的实际价值和未来影响。
📄 摘要（原文）
Self-distillation improves reasoning in large language models by using the model's own rollouts as training signal, typically through implicit logit-level alignment that minimizes KL divergence toward a privileged target distribution. However, because this supervision is generated via uncontrolled sampling, it provides no diagnostic insight into the model's specific errors or corrective guidance for its individual failure patterns. Consequently, the model learns to imitate a privileged distribution rather than receiving fine-grained corrections that pinpoint where and why its reasoning fails. In this paper, we propose Trajectory-Augmented Policy Optimization (TAPO), which advances self-distillation from implicit distributional alignment to explicit trajectory construction. During RL training, the model produces both correct and incorrect rollouts to the same query, and TAPO leverages this contrastive structure to construct micro-reflective corrections, new training trajectories that retain the model's erroneous reasoning up to the point of failure, then insert a natural-language diagnosis and corrected reasoning guided by a correct reference from the same sampling group. Since each trajectory is anchored in the learner's own prefix and solutions, the corrective signal preserves the model's on-policy distribution to a greater extent than the position-wise alignment imposed by KL-based methods. To integrate these trajectories, TAPO introduces difficulty-aware candidate selection at the model's capability boundary and decoupled advantage estimation to prevent gradient contamination. Experiments on AIME 2024, AIME 2025, and HMMT 2025 show that TAPO achieves consistent improvements over GRPO under the same number of training steps. Further analysis demonstrates that TAPO strengthens both first-pass reasoning and error-correction effectiveness.