ShallowDefend: Harnessing Shallow Safety Alignment for Multi-Turn Defender
作者: Qiyao Ma, Xiaodong Liu, Weiwei Yang, Mingqian Feng, Chengquan Guo, Jialin Song, Carolyn Buractaon, Zhe Zhao, Junshan Zhang, Sandeep Atluri, Christopher White, Jianfeng Gao
分类: cs.CL
发布日期: 2026-10-29
💡 一句话要点
提出ShallowDefend以解决多轮对话中的安全对齐问题
🎯 匹配领域: 支柱二：RL算法与架构 (RL & Architecture) 支柱九：具身大模型 (Embodied Foundation Models)
关键词: 安全对齐 多轮对话 强化学习 自然语言处理 模型防御
📋 核心要点
- 现有方法在多轮对话中对有害请求的防御效果较差，安全行为往往集中在前几个输出标记上，导致脆弱性。
- 本文提出Assess → Decide → Respond框架，通过明确的安全决策来引导模型生成回复，从而增强安全性。
- 在两种模型系列和十四个基准测试中，本文方法在安全性上表现最佳，同时在实用性上保持了较高水平。
📝 摘要（中文）
安全对齐使大型语言模型能够可靠地拒绝有害请求，但在多轮对话中，类似请求仍然频繁成功。现有防御方法的不足在于其安全行为主要集中在模型的前几个输出标记上，导致防御效果脆弱。本文提出Assess → Decide → Respond框架，首先评估用户意图，承诺明确的安全决策，然后生成基于该决策的回复。通过强化学习训练承诺决策，实验表明该方法在安全性、实用性和有用性方面表现优异，尤其在多轮对话中实现了最强的安全性。
🔬 方法详解
问题定义：本文旨在解决大型语言模型在多轮对话中对有害请求的防御不足，现有方法的安全行为过于表面化，缺乏深层次的安全决策学习。
核心思路：论文提出的核心思路是通过明确的安全决策来引导模型的回复生成，认为短时间内的决策是影响整个回复轨迹的关键点。
技术框架：整体架构包括三个主要阶段：评估用户意图、承诺安全决策、生成基于该决策的回复。每个阶段都经过精心设计，以确保安全性和实用性。
关键创新：最重要的创新在于将安全决策作为一个标记跨度进行训练，利用自我验证奖励机制，避免了对人工标注的依赖。
关键设计：在参数设置上，采用强化学习来训练决策承诺，损失函数设计为与对话性质相符的规则，确保模型在生成回复时能够遵循安全决策。
🖼️ 关键图片
📊 实验亮点
实验结果显示，ShallowDefend在安全性方面优于现有方法，尤其在多轮对话中，安全性提升幅度显著。通过与两种模型系列的对比，本文方法在安全性、实用性和有用性上均表现出色，确保了用户的安全需求。
🎯 应用场景
该研究的潜在应用领域包括智能客服、社交媒体内容审核和在线教育等场景，能够有效提升多轮对话系统的安全性和用户体验。未来，该方法有望推广到更广泛的自然语言处理任务中，增强模型的安全防护能力。
📄 摘要（原文）
Safety alignment has enabled large language models to reliably refuse harmful requests directly, yet the same request distributed across a multi-turn conversation still succeeds far too often. This brittleness has a mechanistic root: Alignment is shallow, with safety behavior concentrated in a model's first few output tokens, and existing defenses inherit the weakness, supervising the surface form of the response without ever giving the model an explicit safety decision to learn. We take the opposite view of shallowness: if a short span determines the trajectory of an entire response, that span is the highest-leverage place to intervene, and the defender can occupy it first. We propose Assess $\to$ Decide $\to$ Respond, a framework in which the defender assesses the user's intent, commits to an explicit, parseable safety decision, and only then generates a reply conditioned on it, with the commitment trained by reinforcement learning at the payload turn. Because the decision is a token span, the reward can be a rule: a self-verified reward checks the committed label against the conversation's known nature, requiring no judge and no annotation, while the same format also supports an LLM-as-judge reward over a graded decision space with safe redirection. Across two model families and fourteen benchmarks spanning safety, utility, and helpfulness, our defenders achieve the strongest multi-turn safety in a controlled comparison, maintaining their lead in safety while preserving utility at a modest cost to helpfulness. A per-token KL analysis and interventional probes confirm why this works: training concentrates its effect on the brief deliberation that sets a response's trajectory, and the decision committed there is a causal control point. The very shallowness attackers exploit becomes the mechanism of defense.