Shuffled OPSD: When Semantic Consistency Hurts On-Policy Self-Distillation?

📄 arXiv: 2611.04958v1 📥 PDF

作者: Qingyuan Zheng, Zhihao Dou

分类: cs.CL, cs.AI

发布日期: 2026-11-03

备注: 16 pages, 4 figures, 7 tables

💡 一句话要点

提出Shuffled OPSD以优化在政策自蒸馏中的学习效果

🎯 匹配领域: 支柱二：RL算法与架构 (RL & Architecture)

关键词: 自蒸馏 知识蒸馏 语义一致性 推理能力 模型优化 数学推理 深度学习

📋 核心要点

  1. 现有的在政策自蒸馏方法依赖于共享问题的语义结构，可能限制了学习效果。
  2. 论文提出Shuffled OPSD，通过打乱问题陈述的词序，探索其对学习的影响。
  3. 实验结果显示，Shuffled OPSD在多个基准上提升了模型的准确率，尤其在较大模型上表现突出。

📝 摘要（中文）

知识蒸馏通过教师模型的监督提升学生模型的推理能力。在在政策自蒸馏（OPSD）中，学生和教师使用相同的模型，但在不同的上下文中进行操作。学生仅从问题生成推理轨迹，而教师则接收完整的示例并提供逐标记的监督。论文提出Shuffled OPSD，通过打乱共享问题陈述的词序来改善学习效果。实验结果表明，该方法在多个数学推理基准上显著提高了准确率，尤其在Qwen3-1.7B模型上提升了3.52个百分点，挑战了语义一致性在自蒸馏中的普遍有效性。

🔬 方法详解

问题定义：论文要解决的问题是现有的在政策自蒸馏方法过于依赖共享问题的语义一致性，可能导致学习效果不佳。现有方法在教师和学生之间的反馈和生成过程中，语义结构的保持可能并不总是有利于学习。

核心思路：论文的核心思路是引入Shuffled OPSD，通过打乱共享问题的词序来打破语义一致性，从而提高学习效果。这种设计旨在验证语义结构是否总是对学习有益。

技术框架：整体架构包括学生模型生成推理轨迹和教师模型提供监督。Shuffled OPSD在生成过程中对问题陈述进行词序打乱，同时保持教师模型的完整示例不变。

关键创新：最重要的技术创新点在于打乱共享问题的词序，这一方法与传统的保持语义一致性的自蒸馏方法本质上不同，挑战了语义一致性作为普遍有效原则的观点。

关键设计：在实验中，论文对比了不同的词块大小，发现较大的词块并未带来优势，而打乱教师专用示例的词序则在所有规模上都降低了准确率。

📊 实验亮点

实验结果显示，Shuffled OPSD在多个数学推理基准上显著提升了模型的准确率，尤其在Qwen3-1.7B模型上提升了3.52个百分点，优于标准的在政策自蒸馏方法。这些结果表明，打乱共享问题的词序可以有效改善学习效果。

🎯 应用场景

该研究的潜在应用领域包括教育技术、智能辅导系统和自动化推理工具。通过优化自蒸馏过程，能够提升模型在复杂任务中的推理能力，进而改善学习效果，具有重要的实际价值和未来影响。

📄 摘要（原文）

Knowledge distillation improves a student model's reasoning ability through supervision from a teacher. On-policy self-distillation (OPSD) uses the same model as both teacher and student under different contexts: the student generates reasoning trajectories from the question alone, while the teacher additionally receives a complete worked example and provides token-level supervision along those trajectories. Because the problem statement is shared, its content and ordering shape both student generation and teacher feedback. Preserving its semantic structure therefore seems natural, but does it always improve learning? We introduce Shuffled OPSD, which shuffles the words of the shared problem statement while keeping the teacher-only worked example intact. Across three mathematical reasoning benchmarks and three Qwen3 model scales, the method improves mean accuracy over both an intact-target control and Standard OPSD under matched training configurations. The largest gain over Standard OPSD is 3.52 percentage points on Qwen3-1.7B. Further controls show that preserving larger word blocks offers no advantage over word-level shuffling at the evaluated scales, whereas shuffling the teacher-only worked example reduces accuracy at every scale. Cross-problem mixing and target-removal experiments further distinguish the roles of problem content, word order, and teacher context. These results show that disrupting the shared problem statement can improve learning, while disrupting the teacher-only reference hurts performance. They challenge semantic coherence as a universally beneficial principle for OPSD and motivate closer study of how input structure shapes student generation and teacher supervision.