Cost-Efficient Large Language Model Serving for Multi-turn Conversations with CachedAttention
作者: Bin Gao, Zhuomin He, Puru Sharma, Qingxuan Kang, Djordje Jevdjic, Junbo Deng, Xingkun Yang, Zhou Yu, Pengfei Zuo
分类: cs.CL, cs.LG
发布日期: 2024-03-23 (更新: 2024-06-30)
备注: Accepted to USENIX Annual Technical Conference (ATC) 2024
💡 一句话要点
提出CachedAttention以解决多轮对话中的高成本问题
🎯 匹配领域: 支柱九：具身大模型 (Embodied Foundation Models)
关键词: 大型语言模型 多轮对话 注意力机制 KV缓存 推理优化 性能提升 智能客服
📋 核心要点
- 现有的LLM服务在多轮对话中效率低下，需重复计算历史令牌的KV缓存，导致高成本。
- 本文提出CachedAttention机制，重用KV缓存，减少重复计算开销，并优化KV缓存的存储和访问。
- 实验结果显示，CachedAttention将首次令牌响应时间减少87%，提示预填充吞吐量提升7.8倍，推理成本降低70%。
📝 摘要（中文）
与人类进行多轮对话是大型语言模型（LLMs）的基本特性。然而，现有的LLM服务引擎在执行多轮对话时效率低下，因为需要重复计算历史令牌的键值（KV）缓存，导致高昂的服务成本。为了解决这一问题，本文提出了CachedAttention，一种新的注意力机制，能够在多轮对话中重用KV缓存，显著减少重复计算的开销。CachedAttention维护一个分层KV缓存系统，利用成本效益高的内存/存储介质保存所有请求的KV缓存。通过分层预加载和异步保存方案，CachedAttention能够将KV缓存访问与GPU计算重叠，从而减少访问开销。实验结果表明，CachedAttention将首次令牌的响应时间减少了多达87%，提高了多轮对话的提示预填充吞吐量，降低了端到端推理成本高达70%。
🔬 方法详解
问题定义：本文旨在解决现有多轮对话中LLM服务的低效问题，尤其是重复计算历史令牌的KV缓存所带来的高服务成本。
核心思路：提出CachedAttention机制，通过重用KV缓存来减少重复计算开销，同时优化KV缓存的存储和访问策略，以提高多轮对话的响应效率。
技术框架：CachedAttention采用分层KV缓存系统，利用不同存储介质保存KV缓存，并通过分层预加载和异步保存方案来优化缓存访问。整体流程包括KV缓存的存储、访问和更新。
关键创新：CachedAttention的核心创新在于其分层KV缓存管理和调度机制，能够根据推理任务调度器的提示，智能地管理KV缓存的存取，避免缓存失效。
关键设计：设计中采用了层级存储策略、调度器感知的缓存获取与驱逐方案，并通过解耦位置编码和有效截断KV缓存来保持缓存的有效性。具体参数设置和网络结构细节在实验部分进行了详细描述。
🖼️ 关键图片
📊 实验亮点
实验结果表明，CachedAttention机制将首次令牌的响应时间减少了多达87%，提示预填充吞吐量提升至7.8倍，端到端推理成本降低高达70%，展现出显著的性能提升。
🎯 应用场景
该研究的潜在应用领域包括智能客服、虚拟助手和对话系统等，能够显著提升多轮对话的响应速度和效率，降低运营成本，具有广泛的实际价值和未来影响。
📄 摘要（原文）
Interacting with humans through multi-turn conversations is a fundamental feature of large language models (LLMs). However, existing LLM serving engines executing multi-turn conversations are inefficient due to the need to repeatedly compute the key-value (KV) caches of historical tokens, incurring high serving costs. To address the problem, this paper proposes CachedAttention, a new attention mechanism that enables reuse of KV caches across multi-turn conversations, significantly reducing the repetitive computation overheads. CachedAttention maintains a hierarchical KV caching system that leverages cost-effective memory/storage mediums to save KV caches for all requests. To reduce KV cache access overheads from slow mediums, CachedAttention employs layer-wise pre-loading and asynchronous saving schemes to overlap the KV cache access with the GPU computation. To ensure that the KV caches to be accessed are placed in the fastest hierarchy, CachedAttention employs scheduler-aware fetching and eviction schemes to consciously place the KV caches in different layers based on the hints from the inference job scheduler. To avoid the invalidation of the saved KV caches incurred by context window overflow, CachedAttention enables the saved KV caches to remain valid via decoupling the positional encoding and effectively truncating the KV caches. Extensive experimental results demonstrate that CachedAttention significantly decreases the time to the first token (TTFT) by up to 87%, improves the prompt prefilling throughput by up to 7.8$\times$ for multi-turn conversations, and reduces the end-to-end inference cost by up to 70%.