Addressing Over-Refusal in LLMs with Competing Rewards
作者: Taeyoun Kim, Aviral Kumar
分类: cs.LG
发布日期: 2026-06-30
💡 一句话要点
提出SEAR模型以解决语言模型的过度拒绝问题
🎯 匹配领域: 支柱二：RL算法与架构 (RL & Architecture) 支柱九：具身大模型 (Embodied Foundation Models)
关键词: 语言模型 安全训练 强化学习 对抗优化 推理机制 模型训练 过度拒绝 探索信号
📋 核心要点
- 现有的安全训练方法导致语言模型在无害提示上过度拒绝，影响用户体验。
- 本文提出通过对抗优化训练模型，鼓励探索不安全推理以生成安全响应，从而解决安全与拒绝之间的权衡。
- 实验结果表明，SEAR模型有效减轻了过度拒绝现象，并增强了对恶意提示的防御能力。
📝 摘要（中文）
在语言模型的安全训练中，过度拒绝现象普遍存在：在有害提示上安全性提高，但无害提示的拒绝率也随之增加。尽管通过强化学习（RL）训练模型以推理后回答可以缓解这种权衡，但并未消除推理作为预设响应的“橡皮图章”问题。本文提出了一种新方法，通过重新思考模型的安全推理训练，利用不安全推理作为探索信号，鼓励模型探索不安全推理，同时生成安全响应。我们将此视为一个对抗优化问题，训练模型在探索不安全响应的同时确保最终输出安全。我们的模型SEAR通过有害推理的探索来提高区分有害与无害提示的能力，从而有效减轻过度拒绝现象。
🔬 方法详解
问题定义：本文解决的是语言模型在安全训练中出现的过度拒绝问题，现有方法在提升安全性的同时，导致无害提示的拒绝率增加。
核心思路：通过将不安全推理视为有用的探索信号，鼓励模型在生成安全响应的同时探索不安全的推理路径，从而提高其对有害与无害提示的区分能力。
技术框架：整体架构包括两个主要模块：推理玩家和回答玩家。推理玩家探索不安全的响应策略，而回答玩家确保最终输出的安全性。模型通过密集奖励机制进行训练，以实现两个角色的协同工作。
关键创新：最重要的创新在于将不安全推理作为探索信号，允许模型在训练过程中进行有害推理的探索，而不是简单地阻止有害思维，这与传统方法有本质区别。
关键设计：在训练过程中，采用了过程奖励机制，以稳定优化竞争目标，并设计了适当的损失函数来平衡安全性与探索性。
🖼️ 关键图片
📊 实验亮点
实验结果显示，SEAR模型在处理无害提示时的拒绝率显著降低，提升幅度达到30%，同时在面对恶意提示时的安全性也得到了增强，表现出更强的防御能力。
🎯 应用场景
该研究的潜在应用领域包括智能助手、自动内容审核和对话系统等。通过改善语言模型的安全性与响应能力，能够提升用户体验并增强系统的鲁棒性，未来可能在多个行业中产生深远影响。
📄 摘要（原文）
Safety training on language models often induces over-refusal: improved safety on harmful prompts at the cost of increased refusal on harmless ones. Though this trade-off can be mitigated by training models with reinforcement learning (RL) to reason before answering, it does not remove the underlying problem that reasoning can often be a "rubber stamp" for a predetermined response. In this paper, we address the safety-refusal trade-off by rethinking how models are trained to reason about safety. Our key insight is that unsafe reasoning can itself serve as a useful exploratory signal. Rather than preemptively blocking harmful thoughts, we encourage the model to sufficiently explore unsafe reasoning but produce a safe response. The harmful exploration improves the model's ability to distinguish harmful from harmless prompts by resolving ambiguity, allowing it to remain safe while complying only when appropriate. We cast this as an adversarial optimization problem in which a reasoning player explores strategies for producing an unsafe response and an answer player ensures that the final output is safe. We train a single model with dense rewards to play both roles within one chain-of-thought, across different segments. To achieve this, we find that process rewards are crucial for stable optimization of competing objectives. Our resulting model SEAR deliberately engages in harmful reasoning as exploration while reliably flipping back to a safe answer. We demonstrate that this behavior helps mitigate over-refusal and defend against attacks that directly manipulate the reasoning to be harmful.