Only Accepted Tokens Count: An Energy Accounting of Speculative Decoding Under Latency Targets

📄 arXiv: 2610.40287v1 📥 PDF

作者: Bole Ma, Ayesha Afzal, Jan Eitzinger, Harald Koestler, Gerhard Wellein

分类: cs.DC, cs.AI, cs.LG, cs.PF

发布日期: 2026-10-29

💡 一句话要点

提出能量核算方法以优化推理延迟目标下的语言模型解码

🎯 匹配领域: 支柱九：具身大模型 (Embodied Foundation Models)

关键词: 语言模型 投机解码 能量消耗 延迟目标 深度学习 GPU优化 多语言处理

📋 核心要点

  1. 现有的语言模型解码方法在能量消耗和延迟目标之间存在权衡，导致效率低下。
  2. 论文提出的投机解码方法通过草拟多个令牌并在一次前向传播中验证，旨在提高能量利用率。
  3. 实验结果表明，在不同语言和任务下，投机解码的接受率影响能量节省，且在大批量情况下验证消耗主导能量成本。

📝 摘要（中文）

大型语言模型的服务受到设施功耗的限制。每个令牌的能量消耗决定了数据中心能够服务的令牌数量，而延迟目标则限制了用户接收令牌的速度。论文提出了一种投机解码方法，通过草拟多个未来令牌并在单次前向传播中验证，从而节省能量。研究表明，不同的草拟方案和目标模型会影响接受的令牌数量，进而影响能量节省效果。实验在H100和H200 GPU上进行，结果显示在大批量情况下，验证过程的能量消耗占主导，草拟的节能效果会随着批量增大而减小。

🔬 方法详解

问题定义：论文要解决的问题是如何在满足延迟目标的同时优化大型语言模型的能量消耗。现有方法在解码过程中，每生成一个令牌都需从内存中读取权重，导致能量消耗高且效率低下。

核心思路：论文的核心思路是通过投机解码，草拟多个未来令牌并在单次前向传播中进行验证，从而减少能量消耗。通过这种方式，能够在一定程度上提高解码效率，尤其是在高接受率的情况下。

技术框架：整体架构包括草拟器和验证模块。草拟器负责生成多个候选令牌，而验证模块则在单次前向传播中验证这些候选令牌的有效性。实验在SGLang和CUDA图启用的条件下进行，使用H100和H200 GPU，批量大小达到64。

关键创新：最重要的技术创新点在于通过投机解码方法，能够在不同的任务和语言上动态调整接受率，从而优化能量消耗。这与传统方法的逐个令牌解码方式形成了本质区别。

关键设计：在实验中，采用了不同的草拟方案和目标模型，评估其在不同语言（如英语和低资源语言Romansh）上的表现。关键参数设置包括批量大小和模型类型，这些都会影响最终的能量消耗和解码效率。

📊 实验亮点

实验结果显示，在大批量情况下，验证过程的能量消耗占主导，草拟的节能效果随着批量增大而减小。尤其是在低资源语言上，接受率较低，导致能量每令牌的消耗上升，影响整体性能。

🎯 应用场景

该研究的潜在应用领域包括云计算中的语言模型服务、实时对话系统和多语言翻译等。通过优化能量消耗和延迟，能够提升用户体验并降低运营成本，具有重要的实际价值和未来影响。

📄 摘要（原文）

Large language model serving is limited by facility power. Energy per token determines how many tokens a datacenter can serve, while a latency target bounds how quickly each user receives them. During decode, every weight is read from memory for every generated token. Batching amortizes this read across users but makes no individual user faster. Speculative decoding does: a drafter proposes several future tokens, and the target model verifies them in a single forward pass that reuses the same weight read. Because every drafted token must be verified whether or not it is accepted, the energy speculation saves depends on how many drafted tokens are accepted. This saving is hard to pin down. It varies across target models from different attention families and across drafting schemes, and it depends on the text being served. Every drafter we evaluate accepts fewer tokens on conversation than on code, and every trained drafter accepts fewer on Romansh, a low-resource language, than on English. We measure speculative decoding under production serving conditions (SGLang, CUDA graphs enabled) on H100 and H200 GPUs, at batch sizes up to 64 across 5 corpora. On dense models, verification, not drafting, dominates the energy cost. The saving shrinks as the batch grows, because batching has already shared the weight read that speculation would divide among the accepted tokens. At large batch sizes, block drafters push verification past the balance point between arithmetic and memory traffic into the compute-bound regime, where the saving becomes a loss, and a drafter with low acceptance raises energy per token. Meeting a tighter latency target with speculation then costs additional energy. Larger batches lower energy per token without speculation. Speculation adds to that saving only when acceptance is high, and on the model we tested it leaves little headroom for latency-preserving DVFS.