TailSieve: Partial-Rollout-Guided Tail Routing for LLM Rollouts

📄 arXiv: 2608.22788v1 📥 PDF

作者: Tianqi Xu, Lu Lv, Haoyang Huang, Wenjie Huang, Zhanming Shen, Yuhao Shen, Baolin Zhang, Xinyi Hu, Shuang Ge, Jun Dai, Tianyu Liu, Suorong Yang, Zhikai Li, Ye Bai, Jun Zhang, Lei Chen, Yue Li, Mingchen Wan

分类: cs.AI, cs.LG

发布日期: 2026-08-24

💡 一句话要点

提出TailSieve以解决LLM回滚中的长尾路由问题

🎯 匹配领域: 支柱二：RL算法与架构 (RL & Architecture)

关键词: 长尾路由 大规模语言模型 部分回滚 负载均衡 推理优化

📋 核心要点

  1. 现有的回滚请求路由方法通常均匀分配，导致长尾生成请求在高并发解码批次中占据过多时间，影响整体性能。
  2. TailSieve通过部分回滚信号识别长尾生成请求，并采用分层控制器动态调整长尾组和副本分配，有效提升了路由效率。
  3. 实验结果表明，TailSieve在路由速度上比均匀分组提高了1.67倍，并在推理过程中实现了2.59倍的速度提升。

📝 摘要（中文）

大规模回滚已成为现代LLM系统的核心组成部分，涵盖强化学习后训练、在线蒸馏和重采样评估管道。与在线服务不同，长尾生成请求可能主导整个回滚步骤的完成时间。为此，本文提出TailSieve，一个基于部分回滚的框架，联合控制长尾路由和副本分配。通过部分回滚信号识别候选长尾组，TailSieve实现了高效的路由和负载均衡，最终在路由速度上比均匀分组提升了1.67倍，并在低并发长尾池中实现了更高效的推理。

🔬 方法详解

问题定义：本文旨在解决大规模LLM回滚中长尾生成请求导致的性能瓶颈。现有方法均匀路由长尾请求，未能有效利用资源，造成延迟增加。

核心思路：TailSieve的核心思路是利用部分回滚信号来识别长尾生成请求，并通过分层控制器动态调整长尾组和副本分配，以实现更高效的路由和负载均衡。

技术框架：TailSieve的整体架构包括两个主要模块：部分回滚信号收集模块和分层控制器。前者用于识别长尾请求，后者负责动态调整路由策略和副本分配。

关键创新：TailSieve的主要创新在于结合了长尾隔离和负载均衡的路由策略，使用简单的top-k策略近似最优路由，显著提高了回滚效率。

关键设计：在设计中，TailSieve采用了响应工作历史和测量的并发-吞吐模型来指导副本分配，确保长尾池的低并发性，从而支持更高效的推理过程。

🖼️ 关键图片

fig_0
fig_1
fig_2

📊 实验亮点

TailSieve在实验中表现出色，相比于均匀路由，其路由速度提升了1.67倍，且在低并发长尾池中实现了2.59倍的推理速度提升，显著提高了LLM系统的整体性能。

🎯 应用场景

TailSieve的研究成果可广泛应用于大规模语言模型的推理和训练过程中，尤其是在需要高效处理长尾请求的场景中，如在线对话系统、自动内容生成和个性化推荐等领域。其优化策略将为未来的LLM系统设计提供重要参考。

📄 摘要（原文）

Large-scale rollouts have become a core component of modern LLM systems, spanning reinforcement learning (RL) post-training, on-policy distillation (OPD), and sampling-heavy evaluation pipelines. Unlike online serving, which is typically optimized for request-level latency and throughput, a small number of long-tail generations can dominate the end-to-end makespan of an entire rollout step. In practice, rollout requests are often routed uniformly across replicas, which can place extremely long generations inside high-concurrency decoding batches. To address this, we present TailSieve, a partial-rollout-guided framework that jointly controls tail routing and replica allocation for LLM rollouts. In an idealized setting with known completion lengths, we show that makespan-optimal routing in the long-tail regime combines tail isolation with load balancing, and that a simple top-k policy closely approximates this offline optimum. Leveraging the observation that long-tail prompts tend to remain long-tailed across policy updates, TailSieve uses partial rollouts as a training-free signal for identifying candidate tail groups. A hierarchical controller then jointly adapts the number of isolated groups and the replica split between the tail and bulk pools using collected response-work history and a measured concurrency-throughput model. TailSieve achieves up to 1.67x routing-only speedup over uniform group routing. The resulting low-concurrency tail pool further enables route-specialized speculative decoding with MTP or DFlash, achieving up to 2.59x speedup over uniform routing. Selected prompts are regenerated under the current policy, preserving on-policy generation and avoiding additional routing-induced length bias in steady state.