Are Reasoning Models More Prone to Hallucination?

📄 arXiv: 2505.23646v1 📥 PDF

作者: Zijun Yao, Yantao Liu, Yanxu Chen, Jianhui Chen, Junfeng Fang, Lei Hou, Juanzi Li, Tat-Seng Chua

分类: cs.CL, cs.LG

发布日期: 2025-05-29

💡 一句话要点

探讨推理模型在幻觉现象中的脆弱性

🎯 匹配领域: 支柱二：RL算法与架构 (RL & Architecture) 支柱九：具身大模型 (Embodied Foundation Models)

关键词: 大型推理模型 幻觉现象 冷启动微调 强化学习 事实查询

📋 核心要点

  1. 现有的大型推理模型在事实查询任务中表现不一，存在幻觉现象的严重性差异。
  2. 本文通过全面评估LRMs的幻觉现象，提出冷启动监督微调和可验证奖励强化学习能有效缓解幻觉。
  3. 实验结果表明，后训练流程的不同会显著影响LRMs的幻觉表现，揭示了模型不确定性与事实准确性之间的关系。

📝 摘要（中文）

近年来发展的大型推理模型（LRMs）在复杂任务中展现出强大的链式推理能力。然而，这些模型在事实查询任务中的推理能力是否能有效减少幻觉现象仍存在争议。本文从三个方面探讨了推理模型是否更容易产生幻觉，分析了不同后训练流程对幻觉的影响，并揭示了模型不确定性与事实准确性之间的错位关系，为理解LRMs中的幻觉现象提供了初步认识。

🔬 方法详解

问题定义：本文旨在探讨大型推理模型（LRMs）在事实查询任务中产生幻觉的现象及其原因。现有方法在后训练过程中未能有效减少幻觉，导致性能不稳定。

核心思路：通过对LRMs进行全面评估，分析不同后训练流程对幻觉的影响，提出冷启动监督微调和可验证奖励强化学习作为缓解幻觉的有效策略。

技术框架：研究分为三个主要模块：1) 全面评估LRMs的幻觉现象；2) 行为分析，识别影响事实性的关键认知行为；3) 从模型不确定性角度探讨幻觉机制。

关键创新：本文首次系统性地分析了不同后训练流程对LRMs幻觉现象的影响，揭示了冷启动微调和奖励强化学习的有效性。

关键设计：在实验中，采用了冷启动监督微调和可验证奖励强化学习的组合，分析了Flaw Repetition和Think-Answer Mismatch等认知行为对幻觉的影响。

📊 实验亮点

实验结果显示，采用冷启动监督微调和可验证奖励强化学习的LRMs在SimpleQA基准测试中表现出显著的幻觉减少，相较于仅使用蒸馏和无冷启动微调的模型，幻觉现象减轻了约20%。

🎯 应用场景

该研究为大型推理模型在实际应用中的可靠性提供了重要见解，尤其是在需要高准确性的事实查询任务中。未来可在智能问答系统、自动化客服等领域广泛应用，以提升模型的准确性和用户体验。

📄 摘要（原文）

Recently evolved large reasoning models (LRMs) show powerful performance in solving complex tasks with long chain-of-thought (CoT) reasoning capability. As these LRMs are mostly developed by post-training on formal reasoning tasks, whether they generalize the reasoning capability to help reduce hallucination in fact-seeking tasks remains unclear and debated. For instance, DeepSeek-R1 reports increased performance on SimpleQA, a fact-seeking benchmark, while OpenAI-o3 observes even severer hallucination. This discrepancy naturally raises the following research question: Are reasoning models more prone to hallucination? This paper addresses the question from three perspectives. (1) We first conduct a holistic evaluation for the hallucination in LRMs. Our analysis reveals that LRMs undergo a full post-training pipeline with cold start supervised fine-tuning (SFT) and verifiable reward RL generally alleviate their hallucination. In contrast, both distillation alone and RL training without cold start fine-tuning introduce more nuanced hallucinations. (2) To explore why different post-training pipelines alters the impact on hallucination in LRMs, we conduct behavior analysis. We characterize two critical cognitive behaviors that directly affect the factuality of a LRM: Flaw Repetition, where the surface-level reasoning attempts repeatedly follow the same underlying flawed logic, and Think-Answer Mismatch, where the final answer fails to faithfully match the previous CoT process. (3) Further, we investigate the mechanism behind the hallucination of LRMs from the perspective of model uncertainty. We find that increased hallucination of LRMs is usually associated with the misalignment between model uncertainty and factual accuracy. Our work provides an initial understanding of the hallucination in LRMs.