Diffusion-GR2: Diffusion Generative Reasoning Re-ranker

📄 arXiv: 2607.01170v1 📥 PDF

作者: Zhuoxuan Zhang, Kangqi Ni, Yuhang Chen, Mingfu Liang, Xiaohan Wei, Yunchen Pu, Fei Tian, Chonglin Sun, Frank Shyu, Adam, Song, Sandeep Pandey, Luke Simon, Tianlong Chen, Xi Liu

分类: cs.IR, cs.AI

发布日期: 2026-07-01

备注: Work in progress

💡 一句话要点

提出Diffusion-GR2以解决生成推理重排序的效率问题

🎯 匹配领域: 支柱二：RL算法与架构 (RL & Architecture) 支柱九：具身大模型 (Embodied Foundation Models)

关键词: 生成推理 重排序 块扩散模型 转换微调 基于策略的蒸馏 推荐系统 自然语言处理

📋 核心要点

  1. 现有的生成推理重排序器在推理时速度较慢，导致效率低下，尤其是在处理长序列时。
  2. Diffusion-GR2通过将自回归重排序器转换为块扩散重排序器，采用转换微调和基于策略的蒸馏来解决速度和准确性问题。
  3. 实验结果显示，Diffusion-GR2在亚马逊美容数据集上达到了与自回归重排序器相近的性能，同时显著提高了解码速度。

📝 摘要（中文）

生成推理重排序器通过发出思维链来提高推荐准确性，但推理速度较慢。为了解决这一问题，本文提出了Diffusion-GR2，它将自回归重排序器转换为块扩散重排序器。该方法通过转换微调和基于策略的蒸馏来缩小准确性差距，同时保持推理速度的提升。实验结果表明，Diffusion-GR2在亚马逊美容数据集上接近自回归重排序器的性能，同时在推理输出长度上提高了2.4至3.5倍的解码吞吐量。

🔬 方法详解

问题定义：本文旨在解决生成推理重排序器在推理过程中的速度瓶颈，现有自回归解码器在每个推理令牌上需要进行顺序前向传播，导致推理效率低下。

核心思路：Diffusion-GR2通过块扩散模型并行解码多个位置，减少推理时间，同时通过转换微调和基于策略的蒸馏来保持准确性。

技术框架：该方法包括三个主要阶段：首先是转换微调（CFT），将自回归初始化的扩散模型调整为能够独立去噪答案并生成有效排列；其次是基于策略的蒸馏（OPD），在模型自身解码轨迹上进行监督；最后是强化学习阶段，基于重排序奖励进一步优化模型。

关键创新：Diffusion-GR2的创新在于有效地将自回归重排序器转换为块扩散重排序器，同时通过CFT和OPD缩小了准确性差距，这在现有方法中尚未实现。

关键设计：在设计中，CFT阶段使用自回归教师的密集每令牌目标进行监督，确保模型在推理时能够生成有效的排列，强化学习阶段则通过奖励机制进一步提升模型性能。

🖼️ 关键图片

fig_0
fig_1
fig_2

📊 实验亮点

实验结果表明，Diffusion-GR2在亚马逊美容数据集上达到了与自回归重排序器相近的性能，同时在推理输出长度上提高了解码吞吐量，提升幅度为2.4至3.5倍。这一结果展示了块扩散模型在生成推理任务中的有效性。

🎯 应用场景

Diffusion-GR2的研究成果在推荐系统、信息检索和自然语言处理等领域具有广泛的应用潜力。通过提高生成推理的效率，该方法可以帮助构建更快速、更准确的推荐引擎，提升用户体验。此外，未来的研究可以进一步探索其在其他生成任务中的应用。

📄 摘要（原文）

Generative reasoning re-rankers achieve strong recommendation accuracy by emitting a chain-of-thought before re-ordering a candidate list, but they are slow at inference: an autoregressive (AR) decoder spends one sequential forward pass per reasoning token, and the reasoning trace far exceeds the ranking it produces. To reduce this cost, block-diffusion language models decode many positions in parallel over a few denoising steps and are substantially faster, yet naively converting an AR re-ranker into one opens two accuracy gaps: (1) a structural gap: answer positions are denoised in parallel and scored independently, so the decoder emits invalid rankings (duplicated, dropped, or out-of-set identifiers) that AR avoids through left-to-right masking; and (2) a distributional gap: fine-tuning the converted model on fixed teacher trajectories is off-policy relative to its own decoding at inference, leaving a residual accuracy gap. To close both gaps while keeping the speedup, we propose \textbf{Diffusion-GR2}, a recipe that converts our AR reasoning re-ranker (GR2) into a block-diffusion re-ranker. First, conversion fine-tuning (CFT) adapts the AR-initialized diffusion model to denoise the answer into a valid permutation on its own, without an external constrained decoder. Next, on-policy distillation (OPD) then supervises the model on its own decoded trajectories with dense per-token targets from the AR teacher. Finally, we apply a reinforcement-learning (RL) stage against a re-ranking reward on top of OPD's on-policy policy. Experiments on Amazon Beauty demonstrate that Diffusion-GR2 recovers to near-parity with the AR re-ranker, while block-parallel decoding raises decode throughput by $2.4$--$3.5\times$ at the model's reasoning output length. Ablations show that CFT recovers most of the conversion gap, and that on-policy distillation further closes it to the AR reference.