CompactifAI: Extreme Compression of Large Language Models using Quantum-Inspired Tensor Networks
作者: Andrei Tomut, Saeed S. Jahromi, Abhijoy Sarkar, Uygar Kurt, Sukhbinder Singh, Faysal Ishtiaq, Cesar Muñoz, Prabdeep Singh Bajaj, Ali Elborady, Gianni del Bimbo, Mehrazin Alizadeh, David Montero, Pablo Martin-Ramiro, Muhammad Ibrahim, Oussama Tahiri Alaoui, John Malcolm, Samuel Mugel, Roman Orus
分类: cs.CL, cs.AI, cs.LG, quant-ph
发布日期: 2024-01-25 (更新: 2024-05-13)
备注: 5 pages, 4 figures, 2 tables, and supplementary information of 2 pages and 1 figure. Revised version with new benchmarks for LlaMA2-7B
期刊: Proceedings of the 33rd European Symposium on Artificial Neural Networks, Computational Intelligence and Machine Learning (ESANN 2025), Bruges, Belgium, pp. 531-537, April 2025. ISBN: 9782875870926
💡 一句话要点
提出CompactifAI以解决大语言模型压缩问题
🎯 匹配领域: 支柱二：RL算法与架构 (RL & Architecture) 支柱九：具身大模型 (Embodied Foundation Models)
关键词: 大语言模型 模型压缩 量子启发 张量网络 生成式人工智能 深度学习 参数优化
📋 核心要点
- 现有的压缩方法主要通过减少神经元数量来降低模型体积，但并未证明这是最优策略。
- 本文提出的CompactifAI方法利用量子启发的张量网络，专注于模型的相关性空间，实现更有效的压缩。
- 实验结果显示，CompactifAI结合量化技术显著减少了模型的内存和参数数量，同时提升了训练和推理速度。
📝 摘要（中文）
大型语言模型（LLMs）如ChatGPT和LlaMA在生成式人工智能领域迅速发展，但其庞大的体积带来了训练和推理成本高、能耗大及现场部署受限等挑战。传统的压缩方法如剪枝、蒸馏和低秩近似主要集中在减少网络中的神经元数量，而量化则关注于降低单个权重的数值精度。本文提出了CompactifAI，一种基于量子启发的张量网络的创新LLM压缩方法，聚焦于模型的相关性空间，从而实现更精细和可解释的模型压缩。实验表明，CompactifAI与量化结合可以将LlaMA 7B的内存大小减少93%，参数数量减少70%，训练速度提升50%，推理速度提升25%，且准确率仅下降2%-3%。
🔬 方法详解
问题定义：本文旨在解决大型语言模型的压缩问题，现有方法如剪枝和量化在降低模型体积方面存在局限，尤其是对神经元数量的削减并未被证明是最佳策略。
核心思路：CompactifAI通过量子启发的张量网络，聚焦于模型的相关性空间，从而实现更精细的模型压缩。这种方法允许在保持模型性能的同时，显著减少模型的内存和参数数量。
技术框架：该方法的整体架构包括对模型的相关性分析、张量网络的构建与优化，以及与传统压缩技术的结合。主要模块包括模型分析、张量网络构建和压缩实施。
关键创新：CompactifAI的核心创新在于其基于相关性空间的压缩策略，与传统方法不同，它不单纯依赖于减少神经元数量，而是通过张量网络实现更高效的参数表示。
关键设计：在设计中，采用了量子启发的张量网络结构，结合了层敏感性分析，发现深层网络更适合进行张量网络压缩，同时设置了适当的损失函数以平衡压缩率与模型性能。
🖼️ 关键图片
📊 实验亮点
实验结果显示，CompactifAI与量化结合后，LlaMA 7B的内存大小减少93%，参数数量减少70%，训练速度提升50%，推理速度提升25%，且准确率仅下降2%-3%。这些结果远超现有压缩技术的表现，表明该方法的有效性与创新性。
🎯 应用场景
该研究具有广泛的应用潜力，尤其是在需要高效部署大型语言模型的场景，如移动设备、边缘计算和资源受限的环境。通过有效压缩，能够降低能耗和成本，提高模型的可用性和响应速度，推动生成式人工智能的普及与应用。
📄 摘要（原文）
Large Language Models (LLMs) such as ChatGPT and LlaMA are advancing rapidly in generative Artificial Intelligence (AI), but their immense size poses significant challenges, such as huge training and inference costs, substantial energy demands, and limitations for on-site deployment. Traditional compression methods such as pruning, distillation, and low-rank approximation focus on reducing the effective number of neurons in the network, while quantization focuses on reducing the numerical precision of individual weights to reduce the model size while keeping the number of neurons fixed. While these compression methods have been relatively successful in practice, there is no compelling reason to believe that truncating the number of neurons is an optimal strategy. In this context, this paper introduces CompactifAI, an innovative LLM compression approach using quantum-inspired Tensor Networks that focuses on the model's correlation space instead, allowing for a more controlled, refined and interpretable model compression. Our method is versatile and can be implemented with - or on top of - other compression techniques. As a benchmark, we demonstrate that a combination of CompactifAI with quantization allows to reduce a 93% the memory size of LlaMA 7B, reducing also 70% the number of parameters, accelerating 50% the training and 25% the inference times of the model, and just with a small accuracy drop of 2% - 3%, going much beyond of what is achievable today by other compression techniques. Our methods also allow to perform a refined layer sensitivity profiling, showing that deeper layers tend to be more suitable for tensor network compression, which is compatible with recent observations on the ineffectiveness of those layers for LLM performance. Our results imply that standard LLMs are, in fact, heavily overparametrized, and do not need to be large at all.