Looking in the Mirror: Introspecting Side-Effect Misalignments Induced by Fine-Tuning

📄 arXiv: 2608.04347v1 📥 PDF

作者: Kotaro Yoshida, Laura Gomezjurado Gonzalez, Yukinori Yamamoto, Yuji Naraki, Ryotaro Shimizu, Wenya Wang

分类: cs.LG

发布日期: 2026-08-05

💡 一句话要点

提出侧效应内省以解决微调引发的对齐偏差问题

🎯 匹配领域: 支柱九：具身大模型 (Embodied Foundation Models)

关键词: 微调 对齐偏差 内省学习 安全性 深度学习 模型评估 Delta-Aware适配器

📋 核心要点

  1. 现有方法主要关注在特定行为植入的数据集上进行微调，忽视了微调可能导致的意外对齐偏差问题。
  2. 本文提出侧效应内省，关注微调过程中意外产生的对齐偏移，并构建了相应的数据集以支持该研究。
  3. 实验结果表明，DAIA在处理微调模型时表现优越，能够有效提升内省学习的性能，超越现有适配器。

📝 摘要（中文）

微调使源模型在目标领域获得所需能力和行为，但这一过程可能会降低源模型的对齐属性。现有研究主要关注在特定行为植入的数据集上进行微调，而忽视了实际应用中常见的侧效应偏差问题。为此，本文提出了一个新问题设置——侧效应内省，旨在探讨微调过程中意外产生的对齐偏移，并构建了相应的数据集。此外，提出了Delta-Aware Introspection Adapter（DAIA），该机制旨在显著处理基础模型激活及微调引起的激活差异。实验证明，内省学习能够推广到未见的微调模型和安全类别，且DAIA在性能上优于现有的内省适配器。

🔬 方法详解

问题定义：本文旨在解决微调过程中引发的意外对齐偏差问题。现有方法多集中于特定行为的植入，而未能有效处理微调对模型对齐性的负面影响。

核心思路：提出侧效应内省，关注微调过程中产生的对齐偏移，而非仅仅是植入的行为。通过构建新的数据集，支持对这些偏移的分析与理解。

技术框架：整体架构包括基础模型激活的处理和微调后激活差异的分析。引入Delta-Aware Introspection Adapter（DAIA）作为核心模块，增强对内部模型变化的敏感性。

关键创新：DAIA是本文的主要创新点，它通过显式处理基础模型激活和微调引起的激活差异，提升了内省学习的效果，与现有方法相比具有本质区别。

关键设计：DAIA的设计包括特定的参数设置和损失函数，确保在微调过程中能够有效捕捉到对齐偏移的细微变化。

🖼️ 关键图片

fig_0
fig_1
fig_2

📊 实验亮点

实验结果显示，DAIA在多个未见的微调模型和安全类别中表现出色，显著提升了内省学习的性能。与现有内省适配器相比，DAIA在性能上有明显的提升，具体数据表明其在处理对齐偏移时的有效性。

🎯 应用场景

该研究的潜在应用领域包括安全性敏感的人工智能系统，如自动驾驶、医疗诊断等。通过有效识别和处理微调引发的对齐偏差，可以提升模型在实际应用中的安全性和可靠性，具有重要的实际价值和未来影响。

📄 摘要（原文）

Fine-tuning enables a source model to acquire desired capabilities and behaviors in a target domain while retaining much of its general-purpose competence. However, this adaptation process can also degrade alignment properties that were present in the source model. Recent work has shown that large language models can be trained using LoRA-based modules known as introspection adapters (IAs) to describe behavioral changes induced by fine-tuning. However, existing studies primarily consider settings in which the model is fine-tuned on datasets explicitly designed to implant a specific behavior and is then asked to explain the implanted behavior. This differs from practical deployment scenarios, where the central concern is often side-effect misalignment: unintended degradation of alignment caused by fine-tuning on tasks that are not obviously related to safety or alignment. To bridge this gap, we formulate a novel problem setting called \emph{side-effect introspection}, in which the target of introspection is not a behavior explicitly implanted through fine-tuning, but rather alignment shifts that emerge as unintended side effects, and we construct a dataset for this setting. Furthermore, to enhance sensitivity to internal model changes, we propose the Delta-Aware Introspection Adapter (DAIA), a novel mechanism designed to explicitly process both base-model activations and activation differences induced by fine-tuning. Our empirical evaluation shows that introspection learning generalizes to unseen fine-tuned models and safety categories, and that DAIA consistently outperforms existing introspection adapters.