Privileged, but Biased: How PI-Conditioned Teachers Break Self-Distillation
作者: Sarthak Harne, Chinmay Karkar, Yash Pandya, Ahmed Awadallah, Akshay Nambi
分类: cs.AI, cs.LG
发布日期: 2026-08-05
💡 一句话要点
揭示PI条件教师如何破坏自蒸馏的有效性
🎯 匹配领域: 支柱二：RL算法与架构 (RL & Architecture)
关键词: 自蒸馏 特权信息 模型训练 机器学习 推理能力 任务成功率
📋 核心要点
- 现有自蒸馏方法在复杂任务中表现不佳，尽管在简单任务中取得了成功，验证准确率未能提升。
- 论文提出了PI偏差的概念，强调教师模型的目标受到特定参考解的影响，导致学生模型的推理能力下降。
- 实验结果显示，在多种任务中，逐标记损失下降但验证准确率未改善，揭示了自蒸馏的局限性。
📝 摘要（中文）
自蒸馏（SD）作为一种计算高效的强化学习替代方案，依赖于条件于特权信息（PI）的自教师提供密集的逐标记监督。然而，现有研究主要在简单任务中取得了成功，本文探讨了在更复杂任务中自蒸馏的有效性。研究发现，尽管逐标记损失持续下降，但验证准确率并未改善，甚至通常下降。通过分析，作者提出PI偏差的概念，指出教师的目标被特定参考解所影响，导致学生模型在推理时变得盲目，最终未能提升任务成功率。
🔬 方法详解
问题定义：本文旨在解决自蒸馏在复杂任务中失效的问题，现有方法在简单任务中表现良好，但在更具挑战性的环境中未能有效提升模型性能。
核心思路：作者提出PI偏差的概念，认为教师模型的目标受到特定参考解的影响，导致学生模型在学习过程中忽视了正确性，从而影响推理能力。
技术框架：研究中使用了自蒸馏框架，包含教师模型和学生模型两个主要模块。教师模型基于特权信息生成目标，而学生模型则试图匹配这些目标。
关键创新：最重要的创新在于引入PI偏差评分，量化教师模型的目标偏向性，揭示了自蒸馏在缺乏奖励信号时的局限性。
关键设计：在实验中，设置了不同的损失函数和网络结构，重点关注逐标记损失的变化，分析了低信息标记对模型学习的影响。
🖼️ 关键图片
📊 实验亮点
实验结果表明，在多种任务（如问答、数学、编程等）中，逐标记损失持续下降，但验证准确率未见提升，甚至出现下降，揭示了自蒸馏在复杂任务中的有效性不足。这一发现为未来的研究提供了重要的方向。
🎯 应用场景
该研究的潜在应用领域包括自然语言处理、机器学习模型训练和智能代理系统。通过深入理解自蒸馏的局限性，未来可以设计出更有效的学习算法，提高模型在复杂任务中的表现，推动智能系统的实际应用。
📄 摘要（原文）
Self-distillation (SD) has emerged as a compute-efficient alternative to reinforcement learning with verifiable rewards: a self-teacher, conditioned on privileged information (PI) about the answer such as a reference solution, supplies dense per-token supervision to a student that never sees it. Reported gains, however, come almost exclusively from narrow, low-difficulty settings, leaving open a basic question: as a lone objective, with no reward term, does SD teach anything? We reproduce SDPO's reported gains in its easy setting, then apply the identical setup to difficult tasks and find that it does not. Across question answering, mathematics, coding, and multi-turn agentic tool use, across reasoning modes, model sizes, and forms of PI, and under both the SDPO and OPSD recipes, the per-token loss falls steadily while validation accuracy does not improve and typically degrades. We explain this failure through a single causal chain from the loss to the model it produces. The chain begins with PI bias: having seen one particular reference solution, the teacher's per-token target is pulled toward that trajectory rather than toward correctness in general, an effect we quantify with a PI Bias Score. Trained to match this target everywhere, the student's objective becomes nearly blind to whether a rollout is correct, and the loss it assigns falls mostly on low-information tokens like stopwords, punctuation, uncertainty markers, rather than those that determine the answer; within correct rollouts the exploratory tokens incur the highest divergence, so it penalizes the hesitation that reasoning requires. The result is a flatter, less decisive student that is no better at reasoning: as a lone objective, SD optimizes a signal decoupled from task success.