Self-Distillation Zero: Self-Revision Turns Binary Rewards into Dense Supervision

📄 arXiv: 2604.12002 📥 PDF

作者: Yinghui He, Simran Kaur, Adithya Bhaskar, Yongjin Yang, Jiarui Liu, Narutatsu Ri, Liam Fowl, Abhishek Panigrahi, Danqi Chen, Sanjeev Arora

分类: cs.CL

发布日期: 2026-06-12

💡 一句话要点

提出Self-Distillation Zero以解决稀疏奖励监督问题

🎯 匹配领域: 支柱二：RL算法与架构 (RL & Architecture)

关键词: 自蒸馏 强化学习 稀疏奖励 密集监督 模型训练 自然语言处理 代码生成

📋 核心要点

  1. 现有的后训练方法在使用稀疏的二元奖励时，监督信息不足，导致训练效率低下。
  2. SD-Zero方法通过自蒸馏机制，将修订者的反馈用于生成器的训练，避免了对外部教师的依赖。
  3. 在数学和代码推理基准上，SD-Zero的性能提升至少达到10%，超越了包括拒绝微调在内的多种基线方法。

📝 摘要（中文）

当前可验证设置下的后训练方法主要分为两类：强化学习依赖稀疏的二元奖励，而蒸馏则提供密集的标记级监督。本文提出Self-Distillation Zero（SD-Zero），该方法显著提高了训练样本的效率，无需外部教师或高质量示范。SD-Zero通过训练一个模型同时扮演生成器和修订者的角色，利用修订者的反馈将二元奖励转化为密集的自监督信号。在数学和代码推理基准上，SD-Zero在性能上至少提高了10%，超越了多种强基线。

🔬 方法详解

问题定义：当前的后训练方法在使用二元奖励时，往往只能提供稀疏的监督信息，导致模型训练效率低下，且依赖外部教师或高质量示范的成本高昂。

核心思路：SD-Zero通过自蒸馏机制，将修订者的反馈用于生成器的训练，从而将稀疏的二元奖励转化为密集的自监督信号，提升训练效率。

技术框架：SD-Zero的整体架构包括两个主要模块：生成器和修订者。生成器负责生成初始响应，修订者在此基础上进行改进，并利用二元奖励进行条件化训练。

关键创新：SD-Zero的核心创新在于通过自蒸馏将修订者的标记分布作为生成器的监督信号，实现了从稀疏奖励到密集自监督的转变。与现有方法相比，SD-Zero不再依赖外部教师，显著提高了样本效率。

关键设计：在模型设计上，SD-Zero采用了迭代自演化的策略，允许修订者识别生成器响应中的关键标记，并通过定期的教师同步来提升生成性能。

🖼️ 关键图片

fig_0
fig_1
fig_2

📊 实验亮点

在数学和代码推理基准测试中，SD-Zero的性能提升至少达到10%，超越了包括拒绝微调（RFT）、GRPO和自蒸馏微调（SDFT）在内的多种强基线方法，显示出其在相同问题集和训练样本预算下的显著优势。

🎯 应用场景

该研究的潜在应用领域包括自然语言处理、代码生成和自动化推理等。通过提高模型在稀疏奖励环境下的训练效率，SD-Zero能够在多种实际场景中提供更高质量的输出，具有广泛的实际价值和未来影响。

📄 摘要（原文）

Current post-training methods in verifiable settings fall into two categories. Reinforcement learning (RLVR) relies on binary rewards, which are broadly applicable and powerful, but provide only sparse supervision during training. Distillation provides dense token-level supervision, typically obtained from an external teacher or using high-quality demonstrations. Collecting such supervision can be costly or unavailable. We propose Self-Distillation Zero (SD-Zero), a method that is substantially more training sample-efficient than RL and does not require an external teacher or high-quality demonstrations. SD-Zero trains a single model to play two roles: a Generator, which produces an initial response, and a Reviser, which conditions on that response and its binary reward to produce an improved response. We then perform on-policy self-distillation to distill the reviser into the generator, using the reviser's token distributions conditioned on the generator's response and its reward as supervision. In effect, SD-Zero trains the model to transform binary rewards into dense token-level self-supervision. On math and code reasoning benchmarks with Qwen3-4B-Instruct and Olmo-3-7B-Instruct, SD-Zero improves performance by at least 10% over the base models and outperforms strong baselines, including Rejection Fine-Tuning (RFT), GRPO, and Self-Distillation Fine-Tuning (SDFT), under the same question set and training sample budget. Extensive ablation studies show two novel characteristics of our proposed algorithm: (a) token-level self-localization, where the reviser can identify the key tokens that need to be revised in the generator's response based on reward, and (b) iterative self-evolution, where the improving ability to revise answers can be distilled back into generation performance with regular teacher synchronization. Code:this https URL.