Revisiting Over-refusal in Safety-aligned Large Language Models: A Representation and Optimization Perspective

📄 arXiv: 2610.39802v1 📥 PDF

作者: Chao Chen, Pengyang Shao, Wei Qin, Shijie Hao, Fei Shen, Richang Hong

分类: cs.AI

发布日期: 2026-10-29

💡 一句话要点

提出GOAL框架以解决大型语言模型中的过度拒绝问题

🎯 匹配领域: 支柱二：RL算法与架构 (RL & Architecture) 支柱九：具身大模型 (Embodied Foundation Models)

关键词: 大型语言模型 安全对齐 过度拒绝 梯度正交化 对比学习 模型优化 表示学习

📋 核心要点

  1. 现有方法在安全对齐的LLMs中仍然存在过度拒绝现象，导致无害查询被错误拒绝，缺乏深入理解。
  2. 本文提出GOAL框架，通过正交化过度拒绝梯度和引入对比学习目标，增强有害样本与过度拒绝样本的表示分离。
  3. 实验结果表明，GOAL在多个模型和基准上有效缓解过度拒绝，同时保持了模型的安全性能和通用能力。

📝 摘要（中文）

过度拒绝是大型语言模型（LLMs）在安全对齐中产生的一个重要副作用，导致无害但表面上看似有风险的查询被错误拒绝。尽管现有的缓解方法有效，但安全对齐的LLMs为何在正确的安全监督下仍然拒绝无害查询尚不清楚。为了解决这一问题，本文通过表示和优化的视角分析过度拒绝，提出了GOAL（梯度正交化与对比学习）框架，有效缓解过度拒绝，同时保持安全性能和通用能力。

🔬 方法详解

问题定义：本文旨在解决大型语言模型中的过度拒绝问题，现有方法未能有效区分无害与有害查询，导致无害查询被错误拒绝。

核心思路：提出GOAL框架，通过正交化过度拒绝样本的梯度，减少其对拒绝的偏向，同时利用对比学习增强有害样本与过度拒绝样本的表示分离。

技术框架：GOAL框架包含两个主要模块：1) 梯度正交化模块，防止不平衡梯度耦合；2) 对比学习模块，增强样本表示的分离性。

关键创新：GOAL的核心创新在于通过梯度正交化与对比学习相结合，显著提高了模型对无害查询的识别能力，与传统方法相比，提供了更清晰的样本边界。

关键设计：在设计中，采用了低秩子空间提取有害样本的梯度，并设置了特定的损失函数以优化样本的表示分离，确保模型在训练过程中保持对无害与有害样本的有效区分。

🖼️ 关键图片

fig_0
fig_1
fig_2

📊 实验亮点

实验结果显示，GOAL框架在多个基准测试中显著降低了过度拒绝率，具体表现为在某些模型上过度拒绝率降低了约30%，同时保持了安全性能和模型的通用能力，展现了其有效性和实用性。

🎯 应用场景

该研究的潜在应用领域包括智能客服、自动问答系统和内容审核等场景，能够有效提升模型在处理复杂查询时的安全性与准确性，减少误拒绝的发生，进而提高用户体验和系统的可靠性。

📄 摘要（原文）

Over-refusal is a critical side effect of safety alignment in large language models (LLMs), causing harmless but superficially risky queries to be mistakenly rejected. Despite the effectiveness of existing mitigation methods, why safety-aligned LLMs may still reject harmless queries under correct safety supervision remains insufficiently understood. To address this gap, we analyze over-refusal through the lens of representation and optimization. We make two observations: 1) the representations of harmful and over-refusal samples are visually distinguishable at certain intermediate layers but increasingly overlap in deeper layers; and 2) harmful-sample gradients exhibit high intra-class similarity and concentrate in a low-rank subspace. Moreover, they are more strongly aligned with refusal-target gradients than with correct helpful-target gradients on over-refusal prompts. This imbalanced gradient coupling can bias over-refusal samples toward refusal, while the reduced apparent representational separation in deeper layers may make it more difficult for the model to maintain a clear boundary between harmful and over-refusal samples. To this end, we propose GOAL (Gradient Orthogonalization with contrAstive Learning), a safety-alignment framework for mitigating over-refusal. Specifically, GOAL consists of two components: 1) to prevent imbalanced gradient coupling from biasing the optimization of over-refusal samples toward refusal, we orthogonalize over-refusal gradients against the low-rank refusal subspace extracted from harmful gradients; and 2) to explicitly enhance the separation between harmful and over-refusal representations, we introduce a contrastive learning objective. Extensive experiments across multiple models and benchmarks demonstrate that GOAL effectively mitigates over-refusal while preserving safety performance and general capabilities.