GR2 Technical Report
作者: Yufei Li, Zaiwei Zhang, Mingfu Liang, Kavosh Asadi, Jay Xu, Jimmy Kim, Chongyang Bai, Jieyi Zhang, Hongye Xie, Prachi Agrawal, Dian Yu, Tianyi Chen, Jean-Pascal Billaud, Garret Buell, Sachin Patil, Brooke Bian, Zhou Fang, Kevin Huang, Shiva Sudanagunta, Yuzhen Huang, Emma Lu, Chris O'Brien, Yang Song, Lihong Li, Jacob Tao, Zhicheng Zhu, Chao Li, Gaoxiang Liu, Neil Wu, Li Han, Loki Chen, Ming Lei, Greg Rehm, Siyuan Song, Tianwei Zhang, Li Li, Ketan Singh, Yavuz Yetim, Ilyas Atishev, Satendra Gera, Ashkan Sadeghi, Rachel Yan, Nikko Mizutani, Shuaiwen Wang, Song Yang, Zhijing Li, Jiang Liu, Mengying Sun, Fei Tian, Xiaohan Wei, Chonglin Sun, Shuo Gu, Parish Aggarwal, Kaushik Rangadurai, Zhi Hua, Frank Shyu, Ruchit Sharma, Liyuan Li, Shike Mei, Wenlin Chen, Santanu Kolay, Ben Schulte, Deepak Chandra, Adam, Song, Sandeep Pandey, Xi Liu, Hamed Firooz, Luke Simon
分类: cs.IR, cs.AI
发布日期: 2026-07-05
💡 一句话要点
提出GR2框架以优化工业推荐系统的重排序问题
🎯 匹配领域: 支柱二：RL算法与架构 (RL & Architecture) 支柱九：具身大模型 (Embodied Foundation Models)
关键词: 推荐系统 重排序 大语言模型 强化学习 可验证奖励 工业应用 推理轨迹
📋 核心要点
- 现有推荐系统在重排序阶段的研究相对不足，导致用户体验和参与度未能得到有效提升。
- 本文提出GR2框架，通过中期训练、推理轨迹提炼和强化学习等方法，优化重排序过程。
- GR2在工业规模流量上显著提升了推荐效果，R@1、R@3和N@3分别提高了18.7%、7.1%和9.6%。
📝 摘要（中文）
工业推荐系统通过多阶段漏斗服务数十亿用户，其中重排序步骤对用户参与度和下游性能影响显著。尽管对大语言模型（LLMs）的兴趣日益增长，但在工业应用中仍存在三大缺口：重排序阶段未被充分探索，LLMs通常采用零-shot或监督微调，未充分利用强化学习（RL）的推理能力，以及现有目录使用非语义标识符。为此，本文提出GR2（生成推理重排序器），一个端到端框架，结合了基于语义ID的中期训练、通过目标提示和拒绝采样提炼的推理轨迹，以及针对重排序的RL与可验证奖励。此外，GR2还引入了上下文压缩器和On-Policy Distillation（OPD），以降低训练成本。实验结果表明，GR2在工业流量上相较于传统基线提升了18.7%的R@1、7.1%的R@3和9.6%的N@3。
🔬 方法详解
问题定义：本文旨在解决工业推荐系统中重排序阶段的不足，现有方法多集中于检索和初步排序，导致最终用户体验未能得到优化。
核心思路：GR2框架通过结合中期训练、推理轨迹提炼和强化学习，充分利用大语言模型的推理能力，提升重排序效果。
技术框架：GR2的整体架构包括四个主要模块：基于语义ID的中期训练、推理轨迹提炼、强化学习与可验证奖励设计，以及上下文压缩器。
关键创新：GR2的核心创新在于引入了可验证奖励机制，解决了LLMs在重排序中常见的奖励操控问题，并通过OPD降低了训练成本。
关键设计：在模型设计中，采用了高达99%的唯一性语义ID进行训练，使用目标提示和拒绝采样提炼推理轨迹，并设计了适合重排序的奖励函数，以确保模型在实际应用中的有效性。
🖼️ 关键图片
📊 实验亮点
GR2在工业规模流量上实现了显著的性能提升，R@1提高了18.7%，R@3提升了7.1%，N@3提升了9.6%。这些结果表明，GR2在重排序阶段的优化具有重要的实际意义，能够有效提升用户体验。
🎯 应用场景
GR2框架在工业推荐系统中具有广泛的应用潜力，能够显著提升用户的参与度和满意度，尤其是在电商、内容推荐和广告投放等领域。未来，随着技术的进一步发展，GR2可能会被应用于更复杂的推荐场景，推动个性化推荐的进步。
📄 摘要（原文）
Industrial recommendation systems serve billions of users through a multi-stage funnel -- retrieval, early-stage ranking, and re-ranking -- where the final re-ranking step disproportionately shapes user engagement and downstream performance, particularly for carousel and grid display formats. Despite growing enthusiasm for Large Language Models (LLMs) in recommendation, three gaps hinder industrial adoption: (1) most efforts target retrieval and ranking, leaving re-ranking -- the stage closest to the final user experience -- largely underexplored; (2) LLMs are typically deployed zero-shot or via supervised fine-tuning, underutilizing the reasoning capabilities unlocked by reinforcement learning (RL) on verifiable rewards; (3) deployed catalogs index billions of items with non-semantic identifiers that lie outside any base-LLM vocabulary. We present GR2 (Generative Reasoning Re-Ranker), an end-to-end framework that combines (i) mid-training on semantic IDs produced by a tokenizer with >=99% uniqueness, (ii) reasoning-trace distilled from a stronger teacher via targeted prompting and rejection sampling, and (iii) RL with verifiable rewards purpose-built for re-ranking. To make GR2 resource-viable, we further (iv) introduce a context compressor that amortizes training cost, On-Policy Distillation (OPD) as a scalable alternative to SFT -- which we find collapses at industrial scale -- and reasoning distillation for low-latency serving. GR2 delivers +18.7% R@1, +7.1% R@3, and +9.6% N@3 over legacy baselines on industrial-scale traffic. We further find that reward design is critical in re-ranking: LLMs often hack rewards by preserving the incoming order or exploiting position bias, motivating conditional verifiable rewards as essential industrial components.