Whitened Advantage Summation Policy Optimization: Correlation-Aware Aggregation for Multi-Reward Reinforcement Learning

📄 arXiv: 2611.02609v1 📥 PDF

作者: Olaitan Olaleye, Jayson Meribe, Temidayo Olaleye

分类: cs.LG, cs.AI

发布日期: 2026-11-02

💡 一句话要点

提出WAS-PO以解决多奖励强化学习中的相关性问题

🎯 匹配领域: 支柱二：RL算法与架构 (RL & Architecture)

关键词: 多奖励强化学习 奖励白化 相关性处理 模型训练 自然语言处理

📋 核心要点

  1. 现有方法在处理多个相关奖励时，容易导致共享变异的重复计算，影响学习效果。
  2. WAS-PO通过在加权求和前对奖励进行白化处理，确保权重仅应用于独立的奖励成分，从而优化学习过程。
  3. 实验结果显示，WAS-PO在多个奖励的训练中均优于GDPO，尤其在使用三和四个奖励时，训练奖励提升显著。

📝 摘要（中文）

在语言模型的强化学习中，通常通过对多个奖励进行归一化并加权求和来组合奖励。然而，当奖励之间存在相关性时，重复计算了它们共享的变异。本文提出了一种名为WAS-PO的方法，通过在加权求和前对归一化的优势进行白化处理，以确保所选权重仅应用于独立的奖励成分。实验结果表明，WAS-PO在多个奖励的训练中表现优于传统方法GDPO，尤其在使用三和四个奖励时，训练奖励分别提高了3.8%和3.9%。

🔬 方法详解

问题定义：本文旨在解决在多奖励强化学习中，奖励之间的相关性导致的共享变异重复计算问题。现有方法通过固定权重加和奖励，未能有效处理奖励间的相关性，影响了学习效果。

核心思路：WAS-PO方法通过在加权求和前对归一化的优势进行白化处理，使用奖励相关性矩阵的逆平方根，确保所选权重仅应用于独立的奖励成分，从而优化了奖励组合的有效性。

技术框架：该方法的整体架构包括奖励信号的归一化、白化处理和加权求和三个主要模块。首先对奖励进行归一化，然后通过白化处理消除相关性，最后进行加权求和以得到最终的奖励信号。

关键创新：WAS-PO的主要创新在于引入了奖励白化处理，使得加权求和的权重能够准确反映独立奖励的贡献。这一方法与传统的固定权重求和方法本质上不同，能够有效避免共享变异的重复计算。

关键设计：在WAS-PO中，使用了奖励相关性矩阵的逆平方根进行白化处理，并在加权求和中允许权重为负值。此外，方法还采用了批量归一化替代了逐提示归一化，以提高训练效果。

🖼️ 关键图片

fig_0
fig_1

📊 实验亮点

实验结果表明，WAS-PO在使用三和四个奖励时，训练奖励分别比GDPO提高了3.8%和3.9%。然而，在仅使用两个奖励时，WAS-PO的训练奖励略低于GDPO，提示批量归一化可能是影响结果的因素之一。

🎯 应用场景

该研究的潜在应用领域包括自然语言处理、对话系统和多任务学习等场景，尤其是在需要综合多个奖励信号的强化学习任务中。通过优化奖励组合，WAS-PO有望提升模型的学习效率和性能，推动相关领域的发展。

📄 摘要（原文）

Reinforcement learning for language models often combines several rewards by normalizing each one and adding them with fixed weights. When the rewards are correlated, the variation they share is counted once for every reward that carries it. In terms of the ZCA-decorrelated components of the reward signal, the weighted sum applies weights $Σ^{1/2} w$ rather than the chosen weights $w$, where $Σ$ is the reward correlation matrix. We study WAS-PO, which whitens the normalized advantages with $Σ^{-1/2}$ before adding them, so that the chosen weights apply to those components. We derive the scalarized objective whose gradient it follows when normalization statistics are held fixed, show that its weights on the raw rewards can be negative, give an exact two-reward condition under which the lighter-weighted reward is reversed, and give a loose bound on the error from estimating $Σ$ during training. On tool calling with Qwen2.5-1.5B-Instruct, WAS-PO reached a higher mean training reward than GDPO in all seven seeds with three rewards (3.8% higher) and all three seeds with four (3.9% higher). WAS-PO also replaces GDPO's per-prompt normalization with batch-level normalization, and we did not separate the two changes with three or four rewards. The training-reward gains did not carry over to held-out accuracy, which was slightly lower than GDPO's with three rewards (not significant) and 0.03 lower in absolute terms with four (two seeds evaluated). With two rewards, WAS-PO trailed GDPO in training reward, and a three-seed control weakly suggests that the gap comes from batch-level normalization rather than whitening. In a math setting where the two rewards were nearly identical by construction, WAS-PO drove down the lighter-weighted reward, as the sign condition predicts.