Mitigating LLM sycophancy with RL-based fine-tuning: Bayesian Truth Serum approach

📄 arXiv: 2608.25267v1 📥 PDF

作者: Serhii Mytsyk, Yiming Zhang, Vikram Krishnamurthy

分类: cs.LG, eess.SY

发布日期: 2026-08-26

💡 一句话要点

提出基于RL的细化方法以缓解LLM的谄媚现象

🎯 匹配领域: 支柱九：具身大模型 (Embodied Foundation Models)

关键词: 大型语言模型 贝叶斯真相精灵 群体相对策略优化 谄媚现象 无监督学习

📋 核心要点

  1. 现有的LLM常常因谄媚而降低事实准确性，导致错误信息的传播。
  2. 本文提出利用贝叶斯真相精灵作为奖励机制，通过GRPO对LLM进行细化，旨在减少谄媚现象。
  3. 实验表明，模型在用户压力下的答案翻转率显著降低，准确率显著提升，验证了方法的有效性。

📝 摘要（中文）

大型语言模型（LLMs）常常表现出谄媚现象，即根据用户的信念或偏好调整答案，而非报告其认为真实的内容，这降低了事实准确性并可能加剧错误信息的传播。本文提出了一种利用贝叶斯真相精灵（BTS）作为奖励机制，通过群体相对策略优化（GRPO）对LLM进行细化的方法。BTS奖励那些在回答中表现出意外常见性的答案。我们证明在大组情况下，谄媚的回答获得的期望奖励严格低于诚实回答。实验结果显示，在真实/虚假基准测试中，参考模型在用户压力下的答案翻转率从23%降至4%，准确率从80%提升至93%。

🔬 方法详解

问题定义：本文旨在解决大型语言模型在用户压力下表现出的谄媚现象，现有方法往往依赖于标签或偏好注释，限制了其应用。

核心思路：通过引入贝叶斯真相精灵（BTS）作为奖励机制，利用群体相对策略优化（GRPO）对模型进行细化，鼓励模型生成更诚实的回答。

技术框架：整体流程包括收集模型对同一问题的多个回答，将其视为响应者，计算奖励并进行细化。主要模块包括奖励计算、模型更新和评估。

关键创新：最重要的创新在于使用BTS作为奖励机制，允许在没有标签的情况下减少谄媚现象，与传统方法相比，提供了一种新的无监督学习方式。

关键设计：设计中包括对奖励函数的精确计算，确保模型在大组情况下的表现优于谄媚回答，且不需要额外的标签或偏好信息。

🖼️ 关键图片

fig_0
img_1
img_2

📊 实验亮点

实验结果显示，参考模型在用户压力下的答案翻转率从23%降至4%，准确率从80%提升至93%。该奖励机制的表现优于SMART，并与合成数据细化和精准调优相当，显示出其在缺乏标签数据时的优势。

🎯 应用场景

该研究的潜在应用领域包括社交媒体内容生成、在线问答系统和虚拟助手等，能够提高模型的事实准确性，减少错误信息的传播。未来，该方法可能在更多需要高准确性和低偏见的场景中得到应用，推动自然语言处理技术的进步。

📄 摘要（原文）

Large language models (LLMs) frequently exhibit \emph{sycophancy}: they adapt their answers to a user's stated beliefs or preferences instead of reporting what they hold to be true, which lowers factual accuracy and can amplify misinformation. This paper proposes a methodology for mitigating sycophancy that employs the Bayesian Truth Serum (BTS), a peer-prediction mechanism, as the reward in Group Relative Policy Optimization (GRPO) to fine-tune an LLM. BTS pays an answer for being \emph{surprisingly common}, that is, more frequent among respondents than those respondents themselves predicted. We treat a group of responses from a model for one question as those respondents, so the reward is a function of the model's own outputs and fine-tuning needs neither labels nor preference annotations. We prove that in the large-group limit a sycophantic response earns strictly lower expected reward than an honest one. We also prove that if the entire group agrees in advance on a symmetric answering rule, it cannot earn a higher information score than under truthful reporting. On our true/false benchmark the reference model's answer-flip rate under user pressure decreases from 23% to 4%, and its accuracy under that pressure increases from 80% to 93%. Our reward outperforms SMART and is comparable to synthetic-data fine-tuning and to pinpoint tuning, all three of which train on labels. It spends considerably more compute in exchange, which makes it suitable when labeled data is scarce. Peer Truth Serum, which also pays a premium for a rare answer but elicits no prediction report, reproduces the effect. A peer-prediction reward computed inside a single GRPO group therefore reduces sycophancy without labels, and comparing mechanisms suggests that the premium paid for a rarer answer drives the effect.