CARE: Competence-Aware Reward Shaping for Adaptive Reasoning Length in Video-MLLMs

📄 arXiv: 2606.19927v1 📥 PDF

作者: Chengwen Liu, Hao Peng, Jisheng Dang, Hong Peng, Bin Hu, Tat-Seng Chua

分类: cs.CV

发布日期: 2026-06-18

🔗 代码/项目: GITHUB

💡 一句话要点

提出CARE框架以优化多模态视频推理中的自适应推理长度

🎯 匹配领域: 支柱二：RL算法与架构 (RL & Architecture) 支柱九：具身大模型 (Embodied Foundation Models)

关键词: 多模态视频推理 强化学习 奖励塑形 自适应推理长度 能力感知

📋 核心要点

  1. 现有的强化学习方法在多模态视频推理中缺乏灵活的推理长度控制，导致探索不足和冗余推理。
  2. CARE框架通过能力感知的奖励塑形，动态调整推理长度，优化训练过程中的奖励偏好。
  3. 大量实验表明，CARE在推理准确性和令牌效率上均有显著提升，且训练过程中推理长度呈现倒U型特征。

📝 摘要（中文）

在多模态视频推理中，基于强化学习的方法通常依赖于简单且不灵活的推理长度控制策略，这些策略无法适应模型能力的变化。本文提出了CARE，一个能力感知的奖励塑形框架，用于优化多模态推理中的自适应推理长度。CARE通过对通过率的指数移动平均来维护平滑的能力估计，并利用该估计将训练分为多个阶段，逐步将奖励偏好从探索导向的长推理转向效率导向的简洁推理。实验结果表明，CARE在多个视频推理和视频理解基准上显著提高了推理准确性，稳定了强化学习过程，并显著增强了令牌效率。

🔬 方法详解

问题定义：现有的多模态视频推理方法在推理长度控制上过于简单，无法适应模型能力的变化，导致早期探索不足和后期冗余推理的问题。

核心思路：CARE框架通过维护能力的平滑估计，动态调整训练阶段的奖励偏好，从而实现自适应的推理长度优化。这样的设计使得模型能够在不同能力阶段进行有效的推理。

技术框架：CARE的整体架构包括能力估计模块、奖励塑形模块和训练阶段划分。能力估计模块通过指数移动平均计算通过率，奖励塑形模块则根据能力动态调整奖励。

关键创新：CARE的主要创新在于引入能力感知的奖励塑形机制，避免了冗余推理与任务复杂性之间的混淆，显著提高了推理效率和准确性。

关键设计：CARE通过批量统计数据对推理努力进行归一化，并引入后验放大器来增强对历史困难样本的奖励信号，确保模型在训练过程中能够有效学习。

🖼️ 关键图片

fig_0
fig_1
fig_2

📊 实验亮点

实验结果显示，CARE在多个视频推理基准上显著提高了推理准确性，具体提升幅度达到XX%。同时，CARE在训练过程中实现了更短且更具信息量的推理轨迹，表明其在推理预算分配上的有效性。

🎯 应用场景

CARE框架在多模态视频理解和推理任务中具有广泛的应用潜力，能够提升模型在复杂场景下的推理能力和效率。未来，该方法可扩展到其他领域，如机器人视觉、自动驾驶等，推动智能系统的进一步发展。

📄 摘要（原文）

In multimodal video reasoning, reinforcement learning-based methods typically rely on simplistic and inflexible reasoning-length control strategies that fail to adapt to the model's evolving competence. This mismatch may suppress necessary exploration at early stages, while encouraging redundant reasoning and inefficient decoding once the model becomes more competent. In this paper, we propose CARE, a competence-aware reward shaping framework for adaptive reasoning length optimization in multimodal reasoning. Specifically, CARE maintains a smoothed competence estimate via an exponential moving average of pass rates, and uses it to route training into progressive stages that shift the reward preference from exploration-oriented long-form reasoning to efficiency-oriented concise reasoning. To avoid conflating verbosity with intrinsic task complexity, CARE further normalizes reasoning effort with batch-level statistics, and introduces a posterior amplifier to strengthen reward signals for unexpectedly strong performance on historically difficult samples. The proposed mechanism is seamlessly integrated into the GRPO training pipeline and incurs no additional inference-time overhead. Extensive experiments on multiple video reasoning and general video understanding benchmarks demonstrate that CARE consistently improves reasoning accuracy, stabilizes reinforcement learning, and significantly enhances token efficiency. Moreover, CARE exhibits a characteristic inverted-U trajectory of reasoning length during training, and yields shorter yet more informative reasoning traces at convergence, indicating effective adaptive allocation of reasoning budget. We provide the source code for our proposed CARE framework and experiments at https://github.com/1Pansy/Video-CARE.