Rethinking On-Policy Self-Distillation for Thinking Models
作者: Simran Kaur, Narutatsu Ri, Yinghui He, Liam Fowl, Sanjeev Arora
分类: cs.AI, cs.LG
发布日期: 2026-07-06
💡 一句话要点
重新思考基于策略的自蒸馏以提升思维模型性能
🎯 匹配领域: 支柱二：RL算法与架构 (RL & Architecture)
关键词: 自蒸馏 思维模型 自然语言处理 推理能力 特权上下文 模型训练 性能优化
📋 核心要点
- 现有的自蒸馏方法在长推理轨迹中表现不佳，导致思维模型的性能下降。
- 论文提出了重新审视基于策略的自蒸馏方法，强调特权上下文对学习过程的影响。
- 实验结果显示，特权自蒸馏在五个思维模型上导致平均准确率下降，尤其在长推理预算下效果显著。
📝 摘要（中文）
自蒸馏是一种有前景的自我提升方法，尤其适用于思维模型。本文揭示了在长推理轨迹中，特权自蒸馏反而会导致思维模型性能下降，平均准确率下降高达17%。研究表明，特权上下文会影响学习过程，尤其在高熵分叉位置，导致思维模型在自我修正时表现不佳。这一发现强调了在强思维模型的自蒸馏中需要关注令牌级信号，尤其是在修正和推理步骤中。
🔬 方法详解
问题定义：本文旨在解决特权自蒸馏在长推理轨迹中导致思维模型性能下降的问题。现有方法在特权上下文的使用上存在不足，影响了模型的学习效果。
核心思路：论文的核心思路是通过分析特权上下文对思维模型学习的影响，提出在自蒸馏过程中需要关注令牌级信号，尤其是在修正和推理步骤中。
技术框架：整体架构包括模型训练阶段和推理阶段。在训练阶段，模型通过特权上下文进行自蒸馏；在推理阶段，模型利用推理能力吸收特权信息。
关键创新：最重要的创新点在于揭示了特权自蒸馏在高熵分叉位置的负面影响，这与现有方法的设计理念截然不同，强调了特权上下文对思维模型的学习过程的复杂性。
关键设计：在实验中，特权上下文的设置、损失函数的设计以及模型的网络结构都经过精心调整，以确保在不同推理长度下的性能评估。
🖼️ 关键图片
📊 实验亮点
实验结果显示，特权自蒸馏在五个思维模型上导致平均准确率下降高达17%。特别是在长推理预算下，思维模型的性能损失最为显著，强调了特权上下文对模型学习的复杂影响。
🎯 应用场景
该研究的潜在应用领域包括自然语言处理、智能问答系统和教育技术等。通过优化思维模型的自蒸馏过程，可以提升模型在复杂推理任务中的表现，进而推动相关领域的技术进步和应用落地。
📄 摘要（原文）
Self-distillation is a promising recipe for self-improvement in language models. In this setting, a model can serve as its own teacher when given privileged information, such as a solution to a math problem. This seems especially appealing for thinking models, which can use test-time reasoning to absorb the privileged information. Surprisingly, we show that privileged self-distillation degrades thinking models on long reasoning traces: across five Qwen3 and OLMo thinking models evaluated on AIME24, AIME25, and HMMT25, privileged-context distillation causes a relative drop of up to 17% in avg@16 accuracy. The degradation scales with the amount of privileged context withheld from the student and is most pronounced at long rollout budgets, where thinking models otherwise obtain their largest gains. This failure mode is not specific to self-distillation: on-policy distillation (OPD) improves thinking models, but privileged OPD reverses these gains. Our diagnostics link this failure mode to how privileged teacher context reshapes learning at high-entropy forking positions, where multiple continuations remain plausible and may lead to different reasoning paths. Privileged context lowers fork rates in thinking-model rollouts but not in instruction-model rollouts. This leads to an interesting dichotomy, where privileged context can help instruction-tuned models but hurts stronger thinking models. The effect is visible when the student begins a self-correction branch, where privileged OPD penalizes sampled reconsideration tokens that vanilla OPD supports. Thinking models trained with a privileged teacher produce fewer verification, backtracking, and hedging markers, even after length normalization. These findings indicate that self-distillation for strong thinking models requires attention to token-level signal, especially around correction and reasoning steps.