Lynx: Progressive Speculative Quantization for accelerating KV Transfer in Long-Context Inference

📄 arXiv: 2607.01831v1 📥 PDF

作者: Wenchen Han, Gingfung Matthew Yeung, Marco Barletta, William Toner, Amory Hoste, Adam Barker

分类: cs.DC, cs.LG

发布日期: 2026-07-02

备注: 15 pages, 12 figures. This manuscript was originally submitted to SIGCOMM '26 in February 2026

💡 一句话要点

提出Lynx以解决长上下文推理中的KV传输瓶颈问题

🎯 匹配领域: 支柱九：具身大模型 (Embodied Foundation Models)

关键词: 长上下文推理 键值缓存 量化技术 推理加速 智能代理 检索增强生成 深度学习

📋 核心要点

  1. 现有的KV量化技术在长上下文推理中面临网络延迟和推理精度之间的权衡，无法同时满足这两者的需求。
  2. Lynx系统通过将KV缓存分为高优先级和低优先级流，实现了渐进式的KV传输，允许在接收部分数据后即开始解码。
  3. 实验结果显示，Lynx在TTFT和准确性上均优于现有的8位KV量化方案，展现出显著的性能提升。

📝 摘要（中文）

长上下文推理在大型语言模型服务中日益普遍，尤其是在检索增强生成和智能代理系统中。在分散推理中，这些工作负载需要通过网络传输大量的键值（KV）缓存，解码必须在传输完成后才能开始。尽管现有的KV量化技术能够减少数据量并缓解这一瓶颈，但现有方案未能同时实现低网络延迟和高推理精度。本文提出Lynx系统，通过将KV缓存分为高优先级的Anchor流和低优先级的Residual流，实现渐进式、分流的KV传输。解码在接收Anchor流后开始，并在Residual流传输的同时进行推测，最终通过验证确保与高精度解码等效。实验结果表明，Lynx在多个模型和服务工作负载中，TTFT与激进的4位KV量化相当，同时匹配高精度（BF16）推理的准确性，TTFT比标准8位KV量化提高了1.43倍，准确性提升高达5.1%。

🔬 方法详解

问题定义：本文旨在解决长上下文推理中KV缓存传输的延迟问题。现有方法在传输完成前无法开始解码，导致推理效率低下。

核心思路：Lynx系统挑战了KV缓存作为不可分割单元的假设，利用KV缓存中不同位对注意力计算和推理精度的不同贡献，实现部分使用KV缓存。

技术框架：Lynx将KV缓存分为高优先级的Anchor流和低优先级的Residual流。解码在接收Anchor流后开始，Residual流在此过程中并行传输，最后通过验证确保解码结果的等效性。

关键创新：Lynx的核心创新在于渐进式、分流的KV传输机制，允许在未完全接收KV缓存的情况下进行推理，与传统方法相比显著提高了效率。

关键设计：系统设计中，Anchor流负责传输最重要的位，而Residual流则传输剩余的精度部分。通过这种分流设计，Lynx能够在保持高准确性的同时，显著降低TTFT。

🖼️ 关键图片

img_0
img_1

📊 实验亮点

Lynx在多个模型和服务工作负载中实现了与激进的4位KV量化相当的TTFT，同时匹配高精度（BF16）推理的准确性。相比标准的8位KV量化，TTFT提高了1.43倍，准确性提升高达5.1%，展现出显著的性能优势。

🎯 应用场景

Lynx系统的研究成果在大型语言模型的推理服务中具有广泛的应用潜力，尤其是在需要快速响应的智能代理和检索增强生成任务中。通过提高KV传输效率，Lynx能够支持更复杂的应用场景，提升用户体验和系统性能。

📄 摘要（原文）

Long-context inference is increasingly common in large language model (LLM) serving, driven by retrieval-augmented generation and agentic systems. In disaggregated inference, these workloads require transferring large Key-Value (KV) caches across the network, where decoding cannot begin until the transfer completes. Recent KV quantization techniques reduce data volume and alleviate this bottleneck, but existing schemes fail to achieve both low network-exposed latency and high inference accuracy. We challenge the assumption that the KV cache is an indivisible unit that must be fully received before use. We leverage the observation that different bits in the KV cache contribute unequally to attention computation and inference precision: the most significant bits capture the coarse structure of attention and the least significant bits refine precision. This property enables partial use of the KV cache during decoding. We present Lynx, a system that enables progressive, split-stream KV transfer by partitioning the KV cache into a high-priority Anchor stream carrying the most significant bits and a low-priority Residual stream carrying remaining precision. Decoding begins upon receipt of the Anchor stream and proceeds speculatively while the Residual stream is transferred concurrently, followed by verification that ensures equivalence to higher-precision decoding. Across multiple models and serving workloads, Lynx achieves Time-to-First-Token (TTFT) comparable to aggressive 4-bit KV quantization, while matching the accuracy of high-precision (BF16) inference, improving TTFT over standard 8-bit KV quantization by up to $1.43\times$ and improving accuracy over state-of-the-art by up to $5.1\%$.