SpotServe: Serving Generative Large Language Models on Preemptible Instances

📄 arXiv: 2311.15566v1 📥 PDF

作者: Xupeng Miao, Chunan Shi, Jiangfei Duan, Xiaoli Xi, Dahua Lin, Bin Cui, Zhihao Jia

分类: cs.DC, cs.CL, cs.LG

发布日期: 2023-11-27

备注: ASPLOS 2024

💡 一句话要点

提出SpotServe以解决生成大语言模型的低成本服务问题

🎯 匹配领域: 支柱九：具身大模型 (Embodied Foundation Models)

关键词: 生成大语言模型 可抢占实例 动态并行配置 状态推理恢复 二分图匹配 云计算服务 成本优化

📋 核心要点

  1. 现有方法在服务生成大语言模型时面临高计算和内存需求，导致成本高昂。
  2. SpotServe通过动态调整并行配置和状态推理恢复机制，有效应对实例抢占和工作负载波动。
  3. 实验结果显示，SpotServe在P99尾延迟上提升2.4至9.1倍，节省54%的服务成本。

📝 摘要（中文）

生成大语言模型（LLMs）在计算和内存需求上具有很高的要求，使得其低成本服务面临挑战。本文旨在通过利用现代云中的可抢占GPU实例，降低LLMs的服务成本。SpotServe是首个在可抢占实例上实现的分布式LLM服务系统，提出了多项关键技术以实现快速可靠的服务。首先，SpotServe动态调整LLM的并行配置，以应对实例可用性和工作负载的波动，同时平衡整体吞吐量、推理延迟和成本。其次，迁移实例的任务被建模为二分图匹配问题，使用Kuhn-Munkres算法识别最优迁移计划，最小化通信成本。最后，引入状态推理恢复机制，允许在抢占后以较低成本恢复推理。实验表明，SpotServe在P99尾延迟上比现有最佳系统降低了2.4至9.1倍，并节省了54%的成本。

🔬 方法详解

问题定义：本文旨在解决生成大语言模型在可抢占实例上服务时的高成本和频繁抢占带来的挑战。现有方法未能有效应对实例的动态可用性和工作负载波动，导致服务效率低下。

核心思路：SpotServe的核心思路是通过动态调整并行配置和引入状态推理恢复机制，优化LLM的服务性能，降低成本并提高服务的可靠性。这样的设计使得系统能够在抢占发生时快速恢复，减少服务中断。

技术框架：SpotServe的整体架构包括动态并行配置模块、实例迁移模块和状态推理恢复模块。动态并行配置模块根据实例可用性和工作负载实时调整LLM的并行度，实例迁移模块通过二分图匹配优化迁移过程，状态推理恢复模块则确保在抢占后能够快速恢复推理进度。

关键创新：SpotServe的主要创新在于将实例迁移建模为二分图匹配问题，并使用Kuhn-Munkres算法来优化迁移计划。这一方法显著减少了通信开销，并提高了服务的可靠性。

关键设计：在设计中，SpotServe采用了细粒度的推理进度提交机制，以利用云服务提供的宽限期。此外，动态调整的并行配置考虑了吞吐量、延迟和成本之间的权衡，确保了服务的高效性。

🖼️ 关键图片

fig_0
fig_1
fig_2

📊 实验亮点

实验结果显示，SpotServe在P99尾延迟上比现有最佳LLM服务系统降低了2.4至9.1倍，同时在成本上节省了54%。这些显著的性能提升和成本降低，表明SpotServe在实际应用中的巨大潜力和价值。

🎯 应用场景

SpotServe的研究成果在云计算环境中具有广泛的应用潜力，尤其是在需要高效服务生成大语言模型的场景，如智能客服、内容生成和自然语言处理等领域。通过降低服务成本，SpotServe能够使更多企业和开发者能够负担得起高性能的LLM服务，从而推动相关技术的普及与应用。

📄 摘要（原文）

The high computational and memory requirements of generative large language models (LLMs) make it challenging to serve them cheaply. This paper aims to reduce the monetary cost for serving LLMs by leveraging preemptible GPU instances on modern clouds, which offer accesses to spare GPUs at a much cheaper price than regular instances but may be preempted by the cloud at any time. Serving LLMs on preemptible instances requires addressing challenges induced by frequent instance preemptions and the necessity of migrating instances to handle these preemptions. This paper presents SpotServe, the first distributed LLM serving system on preemptible instances. Several key techniques in SpotServe realize fast and reliable serving of generative LLMs on cheap preemptible instances. First, SpotServe dynamically adapts the LLM parallelization configuration for dynamic instance availability and fluctuating workload, while balancing the trade-off among the overall throughput, inference latency and monetary costs. Second, to minimize the cost of migrating instances for dynamic reparallelization, the task of migrating instances is formulated as a bipartite graph matching problem, which uses the Kuhn-Munkres algorithm to identify an optimal migration plan that minimizes communications. Finally, to take advantage of the grace period offered by modern clouds, we introduce stateful inference recovery, a new inference mechanism that commits inference progress at a much finer granularity and allows SpotServe to cheaply resume inference upon preemption. We evaluate on real spot instance preemption traces and various popular LLMs and show that SpotServe can reduce the P99 tail latency by 2.4 - 9.1x compared with the best existing LLM serving systems. We also show that SpotServe can leverage the price advantage of preemptive instances, saving 54% monetary cost compared with only using on-demand instances.