Activation Outliers Matter: Robust Recovery for Quantized Multimodal LLMs

📄 arXiv: 2608.26581v1 📥 PDF

作者: Tanzila Rahman, Mehran Taghian Jazi, Yunke Peng, Zhuang Ma, Anandharaju Durai Raju, Yao Wang, Xing Huang, Hei Yi Mak, Shadan Golestan, Hoang Le, Yonghan Dong, Wei Guo, Yaoyuan Wang

分类: cs.LG

发布日期: 2026-08-27

备注: 14 Pages, 5 figures, 5 tables

💡 一句话要点

提出残差回退量化以解决多模态大语言模型的激活量化问题

🎯 匹配领域: 支柱九：具身大模型 (Embodied Foundation Models)

关键词: 低比特量化 多模态大语言模型 激活量化 残差回退量化 模型性能提升

📋 核心要点

  1. 现有的低比特量化方法在激活量化方面存在显著性能损失，尤其是4比特量化时。
  2. 提出的残差回退量化（RFQ）框架通过引入辅助量化残差路径来改善激活的保真度，避免了架构修改。
  3. 在Wan2.2和Qwen3-VL上的实验表明，RFQ在多个生成和推理基准中显著提升了性能，缩小了与BF16的差距。

📝 摘要（中文）

低比特量化为降低多模态大语言模型（MLLMs）的计算和内存需求提供了有前景的途径。本文系统研究了不同量化方案在视频生成和推理任务中的表现，发现MXFP8几乎无损，而4比特量化则显著降低性能。通过大量消融实验，识别出激活量化是性能损失的主要来源。为此，提出了残差回退量化（RFQ），该框架通过辅助量化残差路径来补充主超低比特激活表示，显著提高了激活的保真度，同时保持了超低比特计算的效率。实验结果表明，RFQ在多个基准测试中有效恢复了量化损失，缩小了与BF16基线的差距。

🔬 方法详解

问题定义：本文旨在解决多模态大语言模型在低比特量化，尤其是4比特量化时，激活量化导致的显著性能损失。现有方法未能有效应对激活量化的挑战，导致模型性能下降。

核心思路：提出的残差回退量化（RFQ）框架通过引入一个辅助的量化残差路径，来补偿激活量化中的误差，从而提高激活的保真度，同时保持超低比特计算的效率。

技术框架：RFQ框架主要包括两个模块：主超低比特激活表示和辅助量化残差路径。主模块负责生成低比特激活表示，辅助模块则用于捕捉和补偿量化误差。

关键创新：RFQ的核心创新在于通过残差路径显式建模和补偿量化误差，这是与现有方法的本质区别，后者通常忽视了激活量化对性能的影响。

关键设计：RFQ设计中，激活的量化过程采用了轻量级的残差网络结构，损失函数则专注于激活保真度的提升，确保在不增加显著计算开销的情况下，优化模型性能。

🖼️ 关键图片

fig_0
fig_1
fig_2

📊 实验亮点

实验结果显示，RFQ在MXFP4和HiF4量化情况下，显著恢复了性能，缩小了与BF16基线的差距，尤其在多个生成和推理基准测试中，RFQ的表现优于传统的量化方法，提升幅度达到显著水平。

🎯 应用场景

该研究的潜在应用领域包括视频生成、自然语言处理和多模态理解等。通过提高低比特量化模型的性能，RFQ能够在资源受限的环境中实现高效的模型部署，具有重要的实际价值和广泛的应用前景。

📄 摘要（原文）

Low-bit quantization offers a promising avenue for reducing the computational and memory demands of Multimodal Large Language Models (MLLMs). Recent hardware support for low-precision formats, ranging from MXFP8 to ultra-low-bit formats such as MXFP4 and HiF4, has accelerated research into efficient MLLM training and deployment. In this work, we present a systematic study of these quantization schemes in representative MLLMs that span both video generation and reasoning tasks. Our analysis shows that MXFP8 achieves near-lossless performance, whereas aggressive 4-bit quantization leads to significant degradation. Through extensive ablations, we identify activation quantization as the primary source of this performance loss, contributing substantially more than weight quantization. Motivated by this observation, we propose Residual Fallback Quantization (RFQ), a lightweight activation reconstruction framework that supplements the primary ulta-low-bit activation representation with an auxiliary quantized residual pathway. By explicitly modeling and compensating for quantization errors, RFQ improves activation fidelity while preserving the efficiency advantages of ultra-low-bit computation. RFQ requires no architectural modifications and incurs negligible computational overhead. Extensive experiments on Wan2.2 and Qwen3-VL demonstrate that RFQ consistently recovers a substantial portion of the performance lost under the quantization of MXFP4 and HiF4, significantly narrowing the gap to BF16 baselines across both generation and 4 reasoning benchmarks. Our findings establish activation quantization as the dominant bottleneck in ultra-low-bit MLLMs and highlight residual-based activation reconstruction as an effective and practical strategy for robust 4-bit deployment.