Does On-Policy Distillation Really Distill? From Noisy Teacher to Self-Improvement

📄 arXiv: 2608.31046v1 📥 PDF

作者: Yi Ding, Ruqi Zhang

分类: cs.LG, cs.CL

发布日期: 2026-08-31

备注: 20 pages, 12 figures

💡 一句话要点

提出On-Policy自适应方法以解决教师监督噪声问题

🎯 匹配领域: 支柱二：RL算法与架构 (RL & Architecture)

关键词: On-Policy蒸馏 自适应学习 强化学习 教师监督 噪声抑制 模型优化 无监督学习

📋 核心要点

  1. 现有的On-Policy蒸馏方法在教师评分时存在噪声，导致学生策略的改进来源不明确。
  2. 论文提出On-Policy自适应（OPSA）方法，通过熵自适应负优势来优化学习信号，减少对教师的依赖。
  3. 实验表明，OPSA在AIME24基准测试中相较于基线模型提升了35.41分，且在多个任务上表现优越。

📝 摘要（中文）

本文探讨了On-Policy蒸馏（OPD）在强化学习中的应用，指出教师评分的噪声对学生的影响不明确。通过定量分析，发现教师监督存在显著噪声，且噪声随教师规模增加而加剧。令人惊讶的是，学生策略对这些噪声不敏感，表现出相似的收敛性能。研究表明，OPD的有效性主要源于对低对数概率标记的抑制，提出了一种新的无监督方法——On-Policy自适应（OPSA），通过熵自适应负优势来优化学习信号。实验结果显示，OPSA在多个基准测试中显著提升了性能，超越了传统OPD方法。

🔬 方法详解

问题定义：本文旨在解决On-Policy蒸馏（OPD）中教师评分噪声对学生策略学习的影响，现有方法在教师监督下的可靠性尚不明确。

核心思路：论文提出On-Policy自适应（OPSA）方法，利用熵自适应负优势来优化学习信号，减少对教师监督的依赖，主要通过抑制低对数概率标记来实现性能提升。

技术框架：OPSA方法的整体架构包括：首先，识别高熵位置并分配更强的学习信号；其次，抑制尾部标记的概率；最后，均匀分配概率质量到头部标记。

关键创新：OPSA的核心创新在于其无监督学习机制，利用熵自适应负优势替代传统的教师监督，显著提高了学习效率和模型性能。

关键设计：在OPSA中，关键参数包括熵阈值的设置和负优势的计算方式，损失函数设计上强调对高熵位置的强化学习信号，确保模型在训练过程中能够有效抑制低概率标记。

🖼️ 关键图片

fig_0
img_1
img_2

📊 实验亮点

实验结果显示，OPSA在AIME24基准测试中相较于基线模型Qwen3-1.7B提升了35.41分，达到了263%的相对增益，并在所有三个基准测试中将Pass@32的性能提升超过一倍。此外，OPSA在Avg@32上超越了传统的OPD方法16.77分，验证了其有效性。

🎯 应用场景

该研究的潜在应用领域包括强化学习、自然语言处理和机器人控制等。通过减少对教师监督的依赖，OPSA方法可以在数据稀缺或教师评分不可靠的情况下，提升模型的学习效率和性能，具有广泛的实际价值和未来影响。

📄 摘要（原文）

On-policy distillation (OPD) offers dense token-level supervision as an alternative to the sparse outcome-level advantages of reinforcement learning with verifiable rewards (RLVR). However, the teacher scores student-generated trajectories that are inherently off-policy for it, so the reliability of its supervision, and hence the source of the student's improvement, remains unclear. We quantitatively analyze teacher supervision during OPD training and find substantial noise whose prevalence increases with teacher scale. Surprisingly, the student policy is insensitive to such noise, converging to comparable performance regardless of whether noisy supervision is retained or removed. Does OPD distill at all? By analyzing what drives its gains, we find that learning concentrates on low log-probability tokens, and using a single fixed negative advantage matches the performance of teacher-provided ones. This suggests that OPD works largely by suppressing low log-probability tokens, which requires no teacher. These findings motivate On-Policy Self-Adaptation (OPSA), a supervision-free method using entropy-adaptive negative advantages. It assigns stronger learning signals to high-entropy positions, suppressing tail tokens, and evenly redistributing probability mass among head tokens. Compared with the base \texttt{Qwen3-1.7B}, OPSA improves Avg@32 by 35.41 points on AIME24, corresponding to a 263\% relative gain, and more than doubles Pass@32 across all three benchmarks. It also outperforms OPD by 16.77 points in Avg@32 on AIME24. Extensive experiments and analyses across model families and tasks further demonstrate its effectiveness and generalizability.