Kalman Meets Curriculum: Efficient Dynamic Prompt Selection for Adaptive RL Finetuning

📄 arXiv: 2607.27610v1 📥 PDF

作者: Haodong Zhu, Yangyang Ren, Yanjing Li, Sheng Xu, Haiguang Liu, Linlin Yang, Baochang Zhang

分类: cs.LG

发布日期: 2026-07-30

💡 一句话要点

提出KGPS以解决RL微调中的动态提示选择问题

🎯 匹配领域: 支柱二：RL算法与架构 (RL & Architecture) 支柱九：具身大模型 (Embodied Foundation Models)

关键词: 强化学习 提示选择 动态状态估计 卡尔曼滤波 模型微调 推理能力 在线学习

📋 核心要点

  1. 现有的在线提示选择方法在评估准确性和计算效率之间存在权衡，难以适应RL的动态训练过程。
  2. 本文提出的KGPS方法通过动态状态估计来选择提示，能够适应策略漂移，避免了静态假设的局限性。
  3. 实验结果显示，KGPS在多个数学推理基准上相比于强基线减少了83%的回合数，同时提升了平均性能0.12点。

📝 摘要（中文）

强化学习（RL）微调显著提升了大型语言模型（LLMs）的推理能力，但其有效性依赖于为当前策略选择适当难度的提示。现有在线方法面临评估准确但成本高与预测高效但假设静态难度之间的权衡。为此，本文提出了一种基于卡尔曼滤波的提示选择方法（KGPS），将提示选择重新定义为动态状态估计问题。KGPS利用线性-高斯状态空间模型建模每个提示的潜在成功率，并通过卡尔曼滤波器维护对提示难度的高斯后验分布。实验表明，KGPS在多个基准测试中显著提高了最终准确性和回合效率，展示了在线提示选择方法的最新性能。

🔬 方法详解

问题定义：本文旨在解决强化学习微调中提示选择的动态性问题。现有方法在评估准确性与计算效率之间存在权衡，难以适应RL训练过程中的非静态难度变化。

核心思路：KGPS方法将提示选择视为动态状态估计问题，通过建模提示的潜在成功率来适应策略的变化。此设计使得提示选择更加灵活，能够有效应对策略漂移。

技术框架：KGPS的整体架构包括两个主要模块：首先是线性-高斯状态空间模型，用于建模提示的潜在成功率；其次是卡尔曼滤波器，用于维护提示难度的高斯后验分布。

关键创新：KGPS的主要创新在于将提示选择问题转化为动态状态估计，而非静态难度预测。这一方法能够实时适应策略的变化，显著提高了提示选择的效率和准确性。

关键设计：KGPS在设计中引入了过程噪声与策略更新幅度的耦合，使得在策略变化较大时不确定性增加。此外，选择提示时通过最大化后验期望训练效用，优先考虑中等难度的提示，同时自然地重访不确定的提示。

🖼️ 关键图片

fig_0
fig_1
fig_2

📊 实验亮点

在多个数学推理基准测试中，KGPS方法相比于强基线减少了83%的回合数，同时在六个数学推理基准上平均性能提升了0.12点，展示了其在在线提示选择中的领先性能。

🎯 应用场景

KGPS方法在强化学习微调中具有广泛的应用潜力，尤其是在需要动态适应的场景，如智能对话系统、自动化决策支持和复杂任务规划等领域。其高效的提示选择机制能够提升模型的推理能力和训练效率，推动相关技术的实际应用和发展。

📄 摘要（原文）

Reinforcement learning (RL) finetuning significantly enhances the reasoning capabilities of large language models (LLMs), yet its effectiveness critically depends on selecting prompts of appropriate difficulty for the current policy. This is challenging because prompt difficulty evolves throughout training. Existing online methods therefore face a trade-off: evaluation-based approaches are accurate but expensive, while prediction-based approaches are efficient but typically assume stationary difficulty, making them ill-suited to RL's non-stationary training dynamics. To address these issues, we propose a Kalman-Guided Prompt Selection method (KGPS), which reformulates prompt selection as a dynamic state estimation problem rather than static difficulty prediction. KGPS models each prompt's latent success rate in logit space using a linear-Gaussian state-space model, with process noise coupled to the magnitude of policy updates so that uncertainty increases when the policy changes more substantially. A Kalman filter then maintains a calibrated Gaussian posterior over prompt difficulty, and prompts are selected by maximizing a posterior-expected training utility that favors intermediate-difficulty prompts while naturally revisiting uncertain ones. The resulting procedure is adaptive to policy drift and requires no additional rollouts beyond standard policy training. Extensive experiments across mathematics, planning, and geometry reasoning benchmarks, as well as multiple RL algorithms, show that KGPS consistently improves both final accuracy and rollout efficiency over strong baselines, establishing state-of-the-art performance among online prompt selection methods. For example, on DeepSeek-R1-Distill-7B, KGPS uses 83% fewer rollouts than DS while even improving the average performance by 0.12 point across six math reasoning benchmarks.