CQIL: Inference Latency Optimization with Concurrent Computation of Quasi-Independent Layers
作者: Longwei Zou, Qingyang Wang, Han Zhao, Jiangang Kong, Yi Yang, Yangdong Deng
分类: cs.CL
发布日期: 2024-04-10 (更新: 2024-07-04)
备注: ACL 2024
💡 一句话要点
提出CQIL以优化推理延迟问题
🎯 匹配领域: 支柱九：具身大模型 (Embodied Foundation Models)
📋 核心要点
- 现有方法主要集中在减少单层计算延迟，但未能有效解决多层累积延迟的问题。
- 本文提出识别准独立层的思路，使其能够并行计算，从而显著降低推理延迟。
- 实验结果表明，CQIL方法在LLaMA-33B模型上可将推理延迟降低48.3%，性能保持接近。
- method_zh
📝 摘要（中文）
随着大规模语言模型的快速发展，其在自然语言处理任务中的表现达到了前所未有的水平。然而，这些模型的有效性依赖于不断增加的参数数量，导致计算复杂度和推理延迟显著增加，影响用户体验。现有方法如张量并行和量化主要关注单层计算延迟，忽视了层数带来的累积延迟。本文提出了一种新方法，通过识别准独立层并进行并行计算，显著降低推理延迟，同时引入旁路技术以减轻信息损失。实验证明，CQIL方法在LLaMA-33B模型上可将延迟降低多达48.3%，同时保持接近的性能水平。
🖼️ 关键图片
📄 摘要（原文）
The fast-growing large scale language models are delivering unprecedented performance on almost all natural language processing tasks. However, the effectiveness of large language models are reliant on an exponentially increasing number of parameters. The overwhelming computation complexity incurs a high inference latency that negatively affects user experience. Existing methods to improve inference efficiency, such as tensor parallelism and quantization, target to reduce per-layer computing latency, yet overlook the cumulative latency due to the number of layers. Recent works on reducing the cumulative latency through layer removing, however, lead to significant performance drop. Motivated by the similarity of inputs among adjacent layers, we propose to identify quasi-independent layers, which can be concurrently computed to significantly decrease inference latency. We also introduce a bypassing technique to mitigate the effect of information loss. Empirical experiments of the proposed approach on the LLaMA models confirm that Concurrent Computation of Quasi-Independent Layers (CQIL) can reduce latency by up to 48.3% on LLaMA-33B, while maintaining a close level of performance.