HiFloat4 Format for End-To-End Reinforcement Learning Post-Training of Large Language Models

📄 arXiv: 2607.26515v1 📥 PDF

作者: Hei Yi Mak, Shadan Golestan, Hoang Le, Mehran Taghian Jazi, Yunke Peng, Yaoyuan Wang, Yao Wang, Junsong Wang, Tianchi Hu, Fengchen He, Guipeng Hu, Tanzila Rahman, Anandharaju Durai Raju

分类: cs.LG, cs.AI

发布日期: 2026-07-29

💡 一句话要点

提出HiFloat4格式与Rollout Residual Quantization以解决FP4强化学习后训练精度问题

🎯 匹配领域: 支柱二：RL算法与架构 (RL & Architecture) 支柱九：具身大模型 (Embodied Foundation Models)

关键词: 强化学习 后训练 量化 FP4 HiFloat4 Rollout Residual Quantization 大规模语言模型 稀疏修正

📋 核心要点

  1. 现有的FP4强化学习方法在精度上存在显著下降，主要由于回滚激活的量化问题。
  2. 论文提出Rollout Residual Quantization（Rollout-ResQ）作为解决方案，通过稀疏的残差修正来恢复精度。
  3. 实验结果显示，结合HiFloat4格式后，精度差距显著缩小，验证了FP4格式选择对可恢复精度的关键影响。

📝 摘要（中文）

我们首次提出了端到端的FP4强化学习后训练方法，其中的策略和训练过程均在4位精度下运行。系统研究表明，FP4强化学习的主要性能下降源于回滚激活量化，而非训练侧的量化误差。通过引入Rollout Residual Quantization（Rollout-ResQ），我们在FP4回滚矩阵乘法中添加了一个稀疏的残差修正项，显著恢复了因异常值导致的精度损失。结合HiFloat4格式，我们在Qwen2.5-3B和Qwen2.5-Math-7B上将精度差距从4.9%缩小至1.1%，使得完全量化的FP4强化学习接近全精度。

🔬 方法详解

问题定义：论文旨在解决在4位精度下进行强化学习后训练时，因回滚激活量化导致的精度下降问题。现有方法未能有效应对这一挑战，尤其是在处理异常值时。

核心思路：提出Rollout Residual Quantization（Rollout-ResQ），通过在FP4回滚矩阵乘法中添加稀疏的残差修正项，来恢复因量化导致的精度损失。此设计旨在兼顾计算效率与精度恢复。

技术框架：整体架构包括训练策略、回滚策略和Rollout-ResQ模块。训练策略在较高精度下进行，而回滚策略则在FP4精度下执行，Rollout-ResQ负责修正回滚过程中的激活量化误差。

关键创新：Rollout-ResQ是本研究的主要创新点，通过引入稀疏的残差修正机制，显著改善了回滚训练与策略训练之间的匹配问题，区别于传统的预训练修复方法。

关键设计：在Rollout-ResQ中，设计了一个硬件友好的稀疏模式，确保修正项的计算开销最小化。同时，HiFloat4格式的三层分层缩放设计，确保在FP4的紧凑预算下保持分辨率。

🖼️ 关键图片

fig_0
fig_1
fig_2

📊 实验亮点

实验结果表明，结合Rollout-ResQ和HiFloat4格式后，Qwen2.5-3B和Qwen2.5-Math-7B的精度差距从4.9%缩小至1.1%。在开放标准MXFP4上，精度差距也从13.6%缩小至5.3%，显示出FP4格式选择对可恢复精度的关键影响。

🎯 应用场景

该研究的潜在应用领域包括大规模语言模型的后训练和强化学习系统，尤其是在资源受限的环境中。通过提高FP4精度，能够在保持计算效率的同时，提升模型的实际应用性能，具有重要的实际价值和未来影响。

📄 摘要（原文）

We present, to our knowledge, the first end-to-end FP4 RL post-training, in which both the rollout and training policies, including their forward and backward passes, operate at 4-bit precision. A systematic study reveals that the dominant source of degradation in FP4 RL is not training-side quantization error but rollout activation quantization: outliers stretch the dynamic range so far that a large number of activation values underflow to zero under FP4. Counterintuitively, restoring the training policy to higher precision while keeping the rollout in FP4 makes accuracy worse than full FP4 baseline, exposing rollout-training mismatch as the principal failure mode and ruling out standard pretraining-style fixes. We address this with Rollout Residual Quantization (Rollout-ResQ): a single residual correction term constrained to a hardware-friendly sparsity pattern, added only to the FP4 rollout matmul -- a lightweight correction that recovers most of the precision lost to outlier-driven underflow without inflating the rollout's compute footprint. On Qwen2.5-3B and Qwen2.5-Math-7B, Rollout-ResQ paired with the HiFloat4 (HiF4) format -- whose three-level hierarchical scaling preserves resolution under FP4's tight 4-bit budget -- closes the accuracy gap to BF16 from 4.9% to 1.1%, bringing fully quantized FP4 RL within striking distance of full precision. Applied to the open-standard MXFP4, the same recipe narrows the gap from 13.6% to 5.3%, revealing that FP4 format choice is a key factor that determines the ceiling on recoverable accuracy. Together, these results establish HiF4 as the enabling format for end-to-end FP4 RL post-training, and Rollout-ResQ as the activation-side mechanism that makes the gap to BF16 closable.