DynamicPTQ: Mitigating Activation Quantization Collapse via Residual-Stream Dynamics

📄 arXiv: 2606.12487 📥 PDF

作者: Zimo Zhao, Maolin Wang, Bowen Yu, Bowen Liu, Xiao Han, Xiangyu Zhao

分类: cs.LG

发布日期: 2026-06-12

💡 一句话要点

提出DynamicPTQ以解决激活量化崩溃问题

🎯 匹配领域: 支柱九：具身大模型 (Embodied Foundation Models)

关键词: 后训练量化 激活量化 残差流 动态量化 大型语言模型 性能提升 低位量化

📋 核心要点

  1. 现有的后训练量化方法在处理激活量化时面临极端值主导激活范围的问题，导致量化误差放大。
  2. 本文提出DynamicPTQ，通过分析残差流的动态特性，识别量化敏感层并为其分配更高的激活精度，以应对动态量化不稳定性。
  3. 实验结果表明，DynamicPTQ在LLaMA-2和LLaMA-3模型上显著提升了性能，并在吞吐量上实现了1.05至1.07倍的提升。

📝 摘要（中文）

后训练量化（PTQ）对于高效的大型语言模型推理至关重要，但在权重、激活和KV缓存均量化为4位精度时，可靠地量化激活仍然具有挑战性。现有方法主要通过基于变换的平滑技术来缓解激活的极端值问题，但忽视了残差流的跨层动态。本文提出DynamicPTQ，通过识别量化敏感层并为其分配8位激活精度，显著提高了LLaMA-2和LLaMA-3模型在W4A4KV4量化下的困惑度和零-shot QA性能，同时实现了1.05至1.07倍的吞吐量提升。

🔬 方法详解

问题定义：本文旨在解决在后训练量化中，激活量化崩溃的问题，现有方法未能有效处理激活的极端值对量化的影响，导致量化误差显著。

核心思路：论文的核心思路是通过分析残差流的动态变化，识别出对量化敏感的层，并为这些层分配8位的激活精度，以此来减轻动态量化的不稳定性。

技术框架：整体架构包括对网络深度中激活的相位变化进行分析，提出Jump Ratio和Historical Feature SNR来表征残差变化，进而指导量化策略的调整。

关键创新：最重要的创新点在于引入了动态感知的量化策略，突破了静态变换平滑方法的局限，能够更好地应对跨层残差变化带来的量化不稳定性。

关键设计：在具体实现中，DynamicPTQ通过动态识别量化敏感层，设置不同的激活精度，同时与现有的PTQ基线方法（如QuaRot、SpinQuant和FlatQuant）直接集成，确保了高效性和兼容性。

🖼️ 关键图片

fig_0
fig_1
fig_2

📊 实验亮点

实验结果显示，DynamicPTQ在LLaMA-2和LLaMA-3模型上，困惑度和零-shot QA性能均有显著提升，且在W4A4KV4量化下，吞吐量提升达到1.05至1.07倍，展示了其在实际应用中的有效性。

🎯 应用场景

该研究具有广泛的应用潜力，特别是在大型语言模型的推理过程中，能够有效提升模型的性能和效率。未来，DynamicPTQ可以被应用于更多需要低位量化的深度学习任务，推动高效推理技术的发展。

📄 摘要（原文）

Post-training quantization (PTQ) is essential for efficient large language model inference, but reliably quantizing activations remains challenging when weights, activations, and KV caches are all quantized to 4-bit precision. A key difficulty lies in massive activations, whose extreme values dominate the activation range and amplify quantization errors. State-of-the-art methods mainly mitigate massive activations through transformation-based smoothing, such as orthogonal rotations and affine scaling, but overlook the cross-layer dynamics of the residual stream. In this paper, we show that massive activations emerge and disappear in a phase-wise pattern across network depth, triggering large residual changes. These changes cause newly injected layer-wise updates to dominate the 4-bit quantization scale and weaken historical residual information. To characterize this behavior, we introduce Jump Ratio and Historical Feature SNR. This suggests that static transformation-based smoothing cannot fully resolve dynamic quantization instability caused by cross-layer residual changes. Based on this analysis, we propose DynamicPTQ, a Dynamic Post-Training Quantization policy for phase-aware mixed-precision activation quantization. DynamicPTQ identifies quantization-sensitive layers from residual-stream dynamics and assigns 8-bit activation precision only to these layers, while keeping weights, KV caches, and other activations in 4-bit precision. It can be directly integrated with strong PTQ baselines such as QuaRot, SpinQuant, and FlatQuant. Experiments on LLaMA-2 and LLaMA-3 show that DynamicPTQ consistently improves perplexity and zero-shot QA performance under W4A4KV4 quantization, while achieving 1.05 to 1.07 times throughput improvement with modest memory overhead. These results demonstrate a practical path toward robust low-bit LLM inference.