Rethinking Reward Supervision: Rubric-Conditioned Self-Distillation
作者: Siyi Gu, Jialin Chen, Sophia Zhou, Arman Cohan, Rex Ying
分类: cs.AI, cs.CL
发布日期: 2026-06-17
💡 一句话要点
提出基于评分标准的自蒸馏方法以改善推理语言模型训练
🎯 匹配领域: 支柱二：RL算法与架构 (RL & Architecture) 支柱九：具身大模型 (Embodied Foundation Models)
关键词: 推理语言模型 自蒸馏 评分标准 强化学习 科学推理 细粒度反馈
📋 核心要点
- 现有的推理语言模型训练方法依赖于昂贵且可能不准确的链式思维注释，导致学习效果受限。
- 提出的评分标准条件自蒸馏方法通过细粒度的评分标准提供反馈，改善了模型的学习过程。
- 在多项科学推理基准测试中，该方法平均超越了GRPO和OPSD，分别提升了1.0和0.9分。
📝 摘要（中文）
后训练推理语言模型通常依赖于监督蒸馏和强化学习，但现有方法存在一些不足。蒸馏依赖于链式思维注释，这些注释获取成本高且可能存在噪声或不完整性；而强化学习则将反馈压缩为标量信号，难以明确改进方向。本文提出了评分标准条件下的自蒸馏框架，通过结构化的细粒度反馈来指导模型学习，避免将单一参考理由作为唯一监督目标。实验结果表明，该方法在科学推理基准测试中表现优异，超越了现有方法的性能。
🔬 方法详解
问题定义：论文旨在解决现有推理语言模型训练中对链式思维注释的依赖及其带来的噪声和不完整性问题，同时强化学习的标量反馈难以明确改进方向。
核心思路：提出的评分标准条件自蒸馏方法通过引入评分标准作为结构化反馈，提供更细粒度的指导，避免将单一参考理由作为唯一的监督目标。
技术框架：该方法采用两阶段管道，第一阶段生成任务特定的评分标准，第二阶段训练评分标准引导的推理模型。
关键创新：最重要的创新在于将评分标准与自蒸馏结合，提供了比标量奖励优化更细致的信用分配机制，显著提升了推理过程的学习效果。
关键设计：在模型训练中，设计了基于评分标准的损失函数，确保模型能够根据评分标准进行逐步优化，同时在网络结构上进行了调整，以适应评分标准的输入和反馈。
🖼️ 关键图片
📊 实验亮点
实验结果显示，评分标准条件自蒸馏方法在科学推理基准测试中表现优异，平均超越GRPO 1.0分和OPSD 0.9分，证明了该方法在推理过程中的有效性和优势。
🎯 应用场景
该研究的潜在应用领域包括教育技术、智能问答系统和自动化推理工具等。通过提供更细致的反馈机制，模型能够更有效地学习复杂任务，提升推理能力，具有广泛的实际价值和未来影响。
📄 摘要（原文）
Post-training of reasoning language models is commonly driven by supervised distillation and reinforcement learning with verifiable rewards. Distillation often relies on chain-of-thought annotations that are expensive to obtain and may themselves be noisy, incomplete, or partially incorrect; even when the final solution is correct, an imperfect rationale can interfere with learning. Reinforcement learning with verified rewards, on the other hand, typically compresses evaluative feedback into a scalar signal, obscuring which aspects of a response should be improved. We propose \textbf{Rubric-Conditioned Self-Distillation}, a framework that incorporates rubrics as structured, fine-grained feedback for on-policy self-distillation. Our method conditions the teacher model on criterion-level rubrics and uses it to provide token-level guidance on the student's own sampled trajectories. This design avoids treating a single reference rationale as the sole supervision target. Instead, rubrics specify what a strong response should satisfy, enabling more fine-grained credit assignment over the reasoning process than scalar reward optimization. We instantiate this framework with a two-stage pipeline that first learns to generate task-specific rubrics and then trains a rubric-guided reasoner. We evaluate on a diverse suite of science reasoning benchmarks and results show that rubric-conditioned self-distillation effectively converts rubric-level criteria into token-level guidance over the reasoning process, surpassing GRPO by 1.0 points and OPSD by 0.9 points on average.