Compression Trinity: Exploring Sparsity, Quantization, and Low-Rank Approximations for LLM Compression

📄 arXiv: 2608.24070v1 📥 PDF

作者: Mohammad Mozaffari

分类: cs.AI, cs.DC, cs.LG, cs.PF

发布日期: 2026-08-25

备注: PhD thesis, University of Toronto, 2026. 156 pages. Chapters extend MKOR (arXiv:2306.01685), SLoPe (arXiv:2405.16325), OPTIMA (arXiv:2512.13886), PATCH (arXiv:2509.23410), and SLiM (arXiv:2410.09615). Official record: https://utoronto.scholaris.ca/items/2cde1f98-6084-46b9-aae2-dbcd045f1215

💡 一句话要点

提出压缩三位一体框架以提升大语言模型的效率与性能

🎯 匹配领域: 支柱九：具身大模型 (Embodied Foundation Models)

关键词: 大语言模型 模型压缩 稀疏性 量化 低秩近似 训练加速 优化器设计

📋 核心要点

  1. 现有的压缩技术往往孤立应用，导致在准确性和效率之间存在明显的瓶颈。
  2. 本文提出的压缩三位一体框架将稀疏性、量化和低秩近似三者结合，以实现更高效的模型压缩。
  3. 实验结果显示，使用该框架的模型在准确性上提升了5.66%，并在训练速度上实现了1.85倍的加速。

📝 摘要（中文）

大语言模型（LLMs）的计算和环境成本限制了其可扩展部署。传统的压缩技术（稀疏性、量化、低秩近似）通常单独应用，导致准确性与效率之间的权衡。本文提出了“压缩三位一体”框架，将三者联合应用：利用稀疏性减少计算，量化降低内存带宽，低秩近似恢复准确性。通过在优化器和模型架构中应用该框架，显著提升了训练效率和模型性能。实验结果表明，联合应用压缩三位一体对于高效、可扩展的高性能LLMs至关重要。

🔬 方法详解

问题定义：本文旨在解决大语言模型的计算和环境成本问题，现有方法在准确性和效率之间存在明显的权衡，无法有效提升模型性能。

核心思路：提出“压缩三位一体”框架，将稀疏性、量化和低秩近似三种技术联合应用，以实现更高效的模型压缩和性能提升。

技术框架：整体架构包括三个主要模块：稀疏性模块用于减少计算量，量化模块用于降低内存带宽，低秩近似模块用于恢复模型的准确性。通过在优化器和模型架构中应用这些模块，提升训练效率。

关键创新：最重要的创新在于将三种压缩技术联合应用，克服了单独应用时的准确性与效率瓶颈，特别是MKOR和SLoPe等新方法的提出。

关键设计：在MKOR中，通过块对角稀疏性和低秩逆运算来近似曲率，降低了复杂度；SLoPe通过双重修剪反向传播实现N:M稀疏性，使用低秩“懒惰”适配器恢复准确性。

🖼️ 关键图片

img_0
img_1
img_2

📊 实验亮点

实验结果显示，MKOR方法将曲率更新复杂度从O(d^3)降低至O(d^2)，加速收敛达1.85倍；SLoPe通过双重修剪实现训练加速1.25倍；SLiM方法在准确性上提升了5.66%，超越了现有最优方法。

🎯 应用场景

该研究的潜在应用领域包括自然语言处理、机器翻译和对话系统等。通过提升大语言模型的训练效率和性能，能够在资源受限的环境中更好地部署和应用这些模型，推动智能应用的发展。

📄 摘要（原文）

Prohibitive computational and environmental costs impede the scalable deployment of Large Language Models (LLMs). Traditional compression techniques (sparsity, quantization, low-rank approximations) are typically applied in isolation, and each hits an accuracy-efficiency wall. This thesis proposes the "Compression Trinity," a unified framework that applies the three pillars jointly: sparsity to reduce computation, quantization to minimize memory bandwidth, and low-rank approximations to recover accuracy. To accelerate pretraining, we apply the Trinity to the optimizer and model architecture. MKOR approximates curvature via block-diagonal sparsity and low-rank inversion, maintaining numerical stability for quantized states; it reduces curvature update complexity from $O(d^3)$ to $O(d^2)$ and accelerates convergence by up to 1.85x over KFAC. SLoPe accelerates training by up to 1.25x via a double-pruned backward pass for N:M sparsity, using low-rank "lazy" adapters in the final 1% of training to recover accuracy. For post-training compression, OPTIMA stabilizes static masks in a zero-training regime by formulating weight reconstruction as globally optimal column-wise quadratic programs, improving zero-shot accuracy by up to 3.97%. Given a fine-tuning budget, PATCH breaks the ceiling of static masks by learning a dynamic hybrid sparsity ratio between 0% and 50%, yielding up to 1.38x speedups. Finally, SLiM realizes the full Compression Trinity in one shot, using mathematically derived low-rank adapters to recover information lost to quantization and sparsity, improving accuracy by up to 5.66% over state-of-the-art methods and outperforming uncompressed dense models at equal parameter budgets by 0.6%. Together, these results show that jointly applying the Compression Trinity is essential for efficient, scalable, high-performance LLMs.