Low-Dimensional High-Leverage Subspace Optimization: Beyond Full-Parameter Coupled Training for Neural Network Quantization

📄 arXiv: 2608.03919v1 📥 PDF

作者: Peng Xia, Junbiao Pang, Zheng Huang

分类: cs.CV

发布日期: 2026-08-04

备注: 9 pages, 2 figures, 7 tables

💡 一句话要点

提出低维高杠杆子空间优化以解决神经网络量化问题

🎯 匹配领域: 支柱二：RL算法与架构 (RL & Architecture)

关键词: 神经网络量化 低比特量化 归一化仿射参数 深度学习优化 模型压缩

📋 核心要点

  1. 现有的低比特量化方法在紧凑网络上准确性下降严重，主要由于全参数耦合训练忽视了参数的异质性。
  2. 本文提出归一化仿射预处理（NAP），通过冻结主干权重，仅微调仿射参数，优化量化友好性。
  3. 实验结果显示，NAP在低比特量化恢复、重建基础的PTQ和全参数QAT上均表现出显著提升。

📝 摘要（中文）

低比特量化在紧凑网络上面临严重的准确性下降，主要源于忽视参数子空间异质性的全参数耦合训练范式。传统方法采用单一优化策略，PTQ重建固定的预训练模型而未改善量化友好性；QAT则共同更新所有参数，导致主干权重与校准参数之间的梯度耦合。本文识别出归一化仿射参数作为主导量化鲁棒性的低维高杠杆子空间，并提出归一化仿射预处理（NAP）进行针对性子空间优化。实验表明，NAP在ImageNet和CIFAR-100上显著提升了低比特量化的恢复效果，并在重建基础的PTQ中表现优异，超越了饱和的全参数QAT，且调优成本极低。

🔬 方法详解

问题定义：本文旨在解决低比特量化在紧凑网络上导致的准确性下降问题，现有方法如PTQ和QAT存在优化不足和梯度耦合的痛点。

核心思路：通过识别归一化仿射参数为低维高杠杆子空间，提出NAP进行针对性优化，以提升量化鲁棒性。此设计旨在在量化前增强模型的量化友好性。

技术框架：整体流程包括两个主要阶段：对于PTQ，NAP冻结主干权重，仅微调仿射参数；对于QAT，采用交替的QAT-NAP方案，解耦特征学习与数值校准。

关键创新：最重要的创新在于识别归一化仿射参数的作用，并通过NAP实现针对性优化，突破了饱和的全参数耦合训练的性能瓶颈。

关键设计：在PTQ中，NAP在全精度模型上进行微调，使用目标假量化图；在QAT中，采用交替优化策略，确保特征学习与校准的独立性。

🖼️ 关键图片

fig_0
fig_1

📊 实验亮点

实验结果表明，NAP在ImageNet和CIFAR-100数据集上显著恢复了低比特量化的性能，提升幅度超过了传统的全参数QAT，且调优成本几乎可以忽略不计。这一成果展示了针对性低维子空间优化的有效性。

🎯 应用场景

该研究的潜在应用领域包括深度学习模型的量化优化，尤其是在资源受限的设备上，如移动设备和嵌入式系统。通过提升量化友好性，未来可在更广泛的应用场景中实现高效的深度学习推理，降低计算和存储成本。

📄 摘要（原文）

Low-bit quantization suffers severe accuracy degradation on compact networks, rooted in the dominant full-parameter coupled training paradigm that ignores parameter subspace heterogeneity. Their limited feature redundancy leaves little room to absorb quantization errors. Conventional pipelines adopt monolithic optimization: PTQ reconstructs fixed pretrained models without improving inherent quantization friendliness; QAT updates all parameters jointly, suffering from gradient coupling between backbone weights and calibration parameters. In this paper, we identify normalization affine parameters as a low-dimensional high-leverage subspace dominating quantization robustness, and propose Normalization Affine Preconditioning (NAP) for targeted subspace optimization. For PTQ, NAP freezes backbone weights and fine-tunes only affine parameters under the target fake-quantization graph on full-precision models, proactively boosting quantization friendliness before downstream reconstruction. For QAT, we introduce an alternating QAT-NAP schema that decouples feature learning and numerical calibration, breaking the performance ceiling of saturated joint training. Theoretical analysis confirms BN affine parameters fully cancel the channel-wise affine component of quantization distortion, while nonlinear rounding and clipping residuals form the irreducible error boundary; distillation-guided NAP acts as directional flatness optimization, projecting teacher-student logit mismatch onto the restricted subspace. Experiments on ImageNet and CIFAR-100 show NAP recovers severely collapsed low-bit quantization, consistently boosts reconstruction-based PTQ, and outperforms saturated full-parameter QAT with negligible tuning cost. This work reveals the principle of targeted low-dimensional subspace optimization, offering a new perspective beyond full-parameter coupled training for efficient deep learning.