Does Forgetting Transfer Across Modalities? A Real-World Benchmark for Cross-Modal Knowledge Unlearning Evaluation

📄 arXiv: 2608.03791v1 📥 PDF

作者: Chunlin Liu, Junnian Chen, Haitong Jiang, Jianyu Zhao, Yingsen Pang, Jingchen Li, Jiabiao He, Youming Lu, Jinhe Bi, Yuntao Du

分类: cs.AI

发布日期: 2026-08-04

💡 一句话要点

提出UNLINK-VL基准以解决跨模态知识遗忘评估问题

🎯 匹配领域: 支柱九：具身大模型 (Embodied Foundation Models)

关键词: 跨模态学习 知识遗忘 视觉-语言模型 评估基准 人工智能安全 多模态评估

📋 核心要点

  1. 现有研究主要集中在单一模态的知识遗忘，缺乏对跨模态知识遗忘转移的深入探讨。
  2. 本文提出UNLINK-VL基准，通过选择可视化实体和相关事实，评估跨模态知识遗忘的有效性。
  3. 实验结果表明，文本模态的遗忘在视觉和跨模态场景中效果较差，强调了跨模态评估的重要性。

📝 摘要（中文）

视觉-语言模型（VLMs）可能会从其预训练语料中记忆敏感、受版权保护或有害的知识，因此删除这些知识对于构建可信赖的人工智能系统至关重要。现有研究主要集中在单一模态内的遗忘，而跨模态知识遗忘的转移研究仍然不足。为此，本文提出了UNLINK-VL，一个针对VLMs的跨模态知识遗忘的真实世界基准。该基准通过选择可视化的真实世界实体作为遗忘目标，并与相应的图像及来自Wikidata的一跳和多跳事实关联，评估遗忘效果和保留效用。实验结果显示，跨模态转移存在明显的不对称性，强调了跨模态遗忘和评估的必要性。

🔬 方法详解

问题定义：本文旨在解决跨模态知识遗忘的评估问题，现有方法在单一模态内的遗忘效果评估不足，无法有效反映跨模态的遗忘转移情况。

核心思路：提出UNLINK-VL基准，通过选择真实世界的可视化实体作为遗忘目标，结合相应的图像和多种事实，构建一个全面的评估框架。

技术框架：UNLINK-VL基准包含四个互补子集，分别评估目标知识的直接遗忘、通过关系知识的遗忘传播、相关非目标知识的保留以及对语义等价查询的鲁棒性。

关键创新：最重要的创新在于引入跨模态评估机制，揭示了文本模态遗忘在视觉和跨模态场景中的低效性，强调了跨模态知识遗忘的必要性。

关键设计：在实验中，模型在文本和多模态遗忘设置下进行训练，评估遗忘效果和保留效用，采用了多种评估指标以确保全面性。

🖼️ 关键图片

fig_0
fig_1
fig_2

📊 实验亮点

实验结果显示，文本模态的多模态遗忘在文本评估中有效，但在视觉和跨模态场景中效果较差，表明仅依赖单一模态评估可能高估知识遗忘的有效性。具体而言，文本模态的遗忘效果在视觉场景中转移不佳，强调了跨模态评估的重要性。

🎯 应用场景

该研究的潜在应用领域包括人工智能系统的安全性和可信性，尤其是在处理敏感信息和版权内容时。通过有效的知识遗忘机制，能够提高模型的道德标准和法律合规性，未来可能对AI系统的设计和评估标准产生深远影响。

📄 摘要（原文）

Vision-Language Models (VLMs), like Large Language Models (LLMs), may memorize sensitive, copyrighted, or harmful knowledge from their pretraining corpora. Removing such knowledge is essential for building trustworthy AI systems. However, existing studies primarily focus on forgetting within individual modalities. Although recent work has begun to explore cross-modal consistency in unlearning, the cross-modal transfer of real-world knowledge unlearning remains insufficiently studied. To address this gap, we introduce UNLINK-VL, a real-world benchmark for cross-modal knowledge unlearning in VLMs. Under a post-hoc unlearning setting in which the original forget and retain corpora are unavailable, UNLINK-VL selects visually identifiable real-world entities as unlearning targets and associates them with corresponding images and one-hop and multi-hop facts derived from Wikidata. The benchmark comprises four complementary subsets that evaluate direct forgetting of target knowledge, the propagation of forgetting through relational knowledge, the preservation of related non-target knowledge, and robustness to semantically equivalent queries. We train models under text-only and multimodal unlearning settings and evaluate forgetting effectiveness and retained utility across textual, visual, and cross-modal scenarios. Extensive experiments reveal a pronounced asymmetry in cross-modal transfer: multimodal unlearning remains effective under textual evaluation, whereas text-only unlearning transfers poorly to visual and cross-modal scenarios. Meanwhile, the evaluated methods largely preserve the models' general capabilities. These findings demonstrate that relying solely on intra-modal evaluation, particularly text-only evaluation, may substantially overestimate the effectiveness of knowledge unlearning in VLMs, underscoring the need for cross-modal unlearning and evaluation.