Reinforcement Learning with Metacognitive Feedback Elicits Faithful Uncertainty Expression in LLMs

📄 arXiv: 2606.32032v1 📥 PDF

作者: Gabrielle Kaili-May Liu, Avi Caciularu, Gal Yona, Idan Szpektor, Arman Cohan

分类: cs.CL, cs.AI

发布日期: 2026-06-30

备注: Code: https://github.com/yale-nlp/RLMF

💡 一句话要点

提出元认知反馈强化学习以解决LLM不确定性表达问题

🎯 匹配领域: 支柱二：RL算法与架构 (RL & Architecture)

关键词: 元认知 强化学习 大型语言模型 不确定性表达 自我评估 数据选择 忠实校准

📋 核心要点

  1. 现有大型语言模型在元认知能力上存在缺陷，导致其在表达不确定性时表现不佳，影响了模型的可信度。
  2. 本文提出了基于元认知反馈的强化学习（RLMF）机制，通过自我判断性能来优化模型的输出和选择高价值训练数据。
  3. 实验结果显示，RLMF在多种任务上实现了最先进的忠实校准表现，并显著提升了模型自我评估和表达能力。

📝 摘要（中文）

元认知是智能的关键组成部分，指监控和调节自身认知过程的能力。然而，现有的大型语言模型（LLMs）在元认知能力上存在系统性缺陷，如高置信度的幻觉、无法识别知识边界以及错误表达内部不确定性，影响了其可信度和可靠性。本文提出了基于元认知反馈的强化学习（RLMF）机制，通过模型自我判断性能来优化完成排名，并通过元认知数据选择识别高价值训练示例。我们在忠实校准（FC）任务中应用这些创新，结果表明RLMF在多样化任务上实现了最先进的FC表现，同时保持了准确性，并超越了标准强化学习63%。

🔬 方法详解

问题定义：本文旨在解决大型语言模型在不确定性表达方面的不足，现有方法无法有效监控和调节模型的自我判断，导致高置信度的错误输出。

核心思路：通过引入元认知反馈机制，模型能够更准确地评估自身性能，从而优化输出和选择训练数据，提升不确定性表达的准确性。

技术框架：整体框架分为两个阶段：首先使用RLMF机制校准模型自报的置信度，然后通过目标输出编辑将其映射到自然的语言不确定性。

关键创新：RLMF机制的提出是本文的核心创新，与传统的强化学习方法相比，RLMF通过元认知反馈显著提升了模型的自我评估能力和输出质量。

关键设计：在RLMF中，设计了特定的损失函数以优化模型的自我判断，并通过元认知数据选择策略来识别高价值的训练示例，确保模型在学习过程中不断自我改进。

🖼️ 关键图片

fig_0
fig_1
fig_2

📊 实验亮点

实验结果表明，RLMF在多样化任务上实现了最先进的忠实校准表现，超越了标准强化学习63%。此外，RLMF显著提升了模型自我评估和表达能力，展示了其在元认知方面的有效性。

🎯 应用场景

该研究的潜在应用领域包括智能对话系统、自动化内容生成和风险评估等。通过提升模型的元认知能力，可以增强其在复杂任务中的表现，提升用户信任度和系统可靠性，未来可能对人机交互和决策支持系统产生深远影响。

📄 摘要（原文）

Metacognition is a critical component of intelligence that describes the ability to monitor and regulate one's own cognitive processes. Yet LLMs exhibit systemic deficiencies in key metacognitive faculties: they hallucinate with high confidence, fail to recognize knowledge boundaries, and misrepresent their internal uncertainty--undermining trustworthiness and reliability. Since monitoring task performance and adapting behavior accordingly are central to metacognition, we posit that models capable of accurately judging their own performance are better positioned to improve it. We operationalize this idea via two novel mechanisms: reinforcement learning with metacognitive feedback (RLMF), a paradigm to refine completion rankings during preference optimization based on the quality of a model's self-judgments of performance, and metacognitive data selection, which uses similar self-judgments to identify high-value training examples, outperforming naive active learning. We apply these innovations to the problem of faithful calibration (FC), a task that is itself fundamentally metacognitive: the goal is to align expressed with intrinsic uncertainty, difficult even for frontier LLMs. We adopt a two-stage, decoupled approach, first using these methods to calibrate the faithfulness of models' self-reported confidence scores, then mapping to natural, context-adaptable linguistic uncertainty via targeted output editing. Extensive experiments show RLMF achieves generalizable, state-of-the-art FC on diverse tasks while preserving accuracy. Further, RLMF surpasses standard RL by up to 63% while enhancing models' ability to assess and express their own capability limits. This positions RLMF as a promising paradigm to enhance LLM metacognition toward improved abilities and alignment, and suggests metacognitive performance as an effective RL signal to overcome limits of prior intrinsic feedback methods.