RolloutPipe: Overlapping Pipelined Rollout and Training in Disaggregated On-Policy LLM Reinforcement Learning

📄 arXiv: 2606.26997v1 📥 PDF

作者: Rongjian Chen, Jianmin Hu, Kejiang Ye, Minxian Xu

分类: cs.DC, cs.LG

发布日期: 2026-06-25

备注: 15 pages

期刊: Proceedings of the 2026 International Conference on Cognitive Computing (ICCC 2026)

💡 一句话要点

提出RolloutPipe以解决异构RLVR系统中的训练与回滚效率问题

🎯 匹配领域: 支柱二：RL算法与架构 (RL & Architecture) 支柱九：具身大模型 (Embodied Foundation Models)

关键词: 强化学习 大型语言模型 异构计算 资源优化 训练效率 策略训练 回滚生成

📋 核心要点

  1. 现有的同步RLVR系统在回滚完成前无法开始训练，导致资源闲置和效率低下。
  2. RolloutPipe通过完整组管道和前沿组调度技术，实现了训练与回滚的重叠，提高了资源利用率。
  3. 在Qwen3-1.7B模型的实验中，RolloutPipe将回滚到训练结束的时间缩短了30.7%-42.3%，显著提升了效率。

📝 摘要（中文）

大型语言模型（LLM）后训练中的推理越来越依赖于可验证奖励的强化学习（RLVR），模型通过真实反馈学习数学、逻辑和科学任务。现代RLVR系统采用解耦架构，独立的GPU池进行回滚生成和策略训练。然而，现有的同步策略在回滚完成前无法开始训练，导致训练GPU池闲置。虽然异步管道可以重叠这两个阶段，但会导致训练使用过时数据。为了解决这些问题，本文提出了RolloutPipe框架，通过完整组管道和前沿组调度技术，实现了训练与回滚的重叠，显著提高了效率。实验结果显示，RolloutPipe在多个基准测试中将回滚到训练结束的时间缩短了30.7%-42.3%。

🔬 方法详解

问题定义：本文旨在解决现有同步RLVR系统在回滚生成与策略训练之间的效率低下问题，尤其是在回滚完成前训练GPU池的闲置现象。

核心思路：RolloutPipe通过将固定权重的回滚转变为完整组管道，使得可训练组在后续组生成时就能开始训练，从而实现训练与回滚的重叠。

技术框架：RolloutPipe的整体架构包括两个主要技术：完整组管道（CGP）和前沿组调度（FGD）。CGP在组生成完成后立即将可训练组发送到训练器，而FGD则优先接收形成下一个训练批次所需的前沿组请求。

关键创新：RolloutPipe的创新在于其设计允许在回滚未完成时开始训练，保持了策略的正确性，并有效减少了训练等待时间。与现有方法相比，它实现了更高的资源利用率和更快的训练速度。

关键设计：在技术细节上，RolloutPipe采用FIFO调度策略，确保训练器接收的组是准备好的，并通过前沿组调度策略优化了训练批次的形成。

🖼️ 关键图片

fig_0
fig_1
fig_2

📊 实验亮点

RolloutPipe在Qwen3-1.7B模型的实验中，成功将回滚到训练结束的时间缩短了30.7%-42.3%，并降低了训练器的等待比例37%-76%。与现有的Slime系统相比，RolloutPipe展现了显著的性能提升，证明了其在异构RLVR系统中的有效性。

🎯 应用场景

RolloutPipe的研究成果在大型语言模型的后训练阶段具有广泛的应用潜力，尤其是在需要高效资源利用的异构计算环境中。其设计理念可以推广到其他强化学习任务中，提升训练效率和模型性能，具有重要的实际价值和未来影响。

📄 摘要（原文）

Large language model (LLM) post-training for reasoning increasingly relies on reinforcement learning with verifiable rewards (RLVR), where models learn from ground-truth feedback on mathematical, logical, and scientific tasks. To enable flexible resource allocation and support heterogeneous training setups, modern RLVR systems adopt disaggregated architectures that decouple rollout generation and policy training across independent GPU pools. However, existing synchronous on-policy GRPO (Group Relative Policy Optimization) RLVR systems finish an entire rollout before starting training, leaving the trainer GPU pool idle while rollout is still ongoing. Asynchronous RL pipelines overlap the two stages, but at the cost of training on stale data. To address these challenges, we propose RolloutPipe, a post-training framework for disaggregated RLVR systems, which turns the fixed-weight rollout into a complete-group pipeline where trainable groups move to the trainer while later groups are still being generated. RolloutPipe achieves this through two techniques including complete-group pipelining (CGP) and frontier-group dispatch (FGD). CGP dispatches each trainable complete group to the trainer FIFO as soon as group materialization finishes, and FGD is an admission policy on the Rollout node that first admits requests for the frontier groups needed to form the next training batch, so that trainer-ready groups arrive earlier and more steadily. The design starts training before the rollout completes while maintaining on-policy correctness. Evaluated on Qwen3-1.7B across four reasoning and science benchmarks and twelve rollout settings, RolloutPipe shortens the rollout-to-train-end time by 30.7%-42.3%, and lowers the trainer waiting ratio by 37%-76% compared to Slime, a state-of-the-art rollout and training system.