Dynamic Gated Cross-Modal Fusion with Sarcastic-aware Contrastive Regularization for Multimodal Sarcasm Detection

📄 arXiv: 2608.19942v1 📥 PDF

作者: Hao Guo, Subin Huang, Junjie Chen, Zhifa Geng, Sanmin Liu, Chao Kong

分类: cs.CL

发布日期: 2026-08-20

备注: Accepted to SEKE 2026. 6 pages, 3 figures

💡 一句话要点

提出动态门控跨模态融合以解决多模态讽刺检测问题

🎯 匹配领域: 支柱九：具身大模型 (Embodied Foundation Models)

关键词: 多模态讽刺检测 动态门控 跨模态融合 对比正则化 深度学习

📋 核心要点

  1. 现有方法在多模态讽刺检测中存在固定融合策略，难以捕捉细微的讽刺线索和实例特定的模态交互。
  2. 本文提出的框架通过动态门控跨模态融合和讽刺感知对比正则化，能够自适应地调整文本和视觉模态的贡献。
  3. 在MMSD和MMSD2.0数据集上的实验结果表明，所提方法在性能上显著优于强基线，验证了其有效性。

📝 摘要（中文）

多模态讽刺检测旨在从多模态内容中识别讽刺意图，其中字面意义与上下文线索之间的不一致通常暗示讽刺。尽管该任务受到越来越多的研究关注，但由于实例依赖的模态贡献和误导性的语义一致性，准确检测仍然具有挑战性。现有方法通常依赖固定的融合策略，将讽刺视为通用的跨模态不匹配，限制了其捕捉细微讽刺线索和实例特定模态交互的能力。为了解决这些挑战，本文提出了一种新颖的多模态讽刺检测框架，集成了动态门控跨模态融合与讽刺感知对比正则化（SaCR）。

🔬 方法详解

问题定义：本文旨在解决多模态讽刺检测中的模态贡献不一致和误导性语义一致性问题。现有方法往往无法有效捕捉细微的讽刺线索，导致检测准确性不足。

核心思路：提出的框架通过动态门控机制自适应地调整不同模态的贡献，结合讽刺感知对比正则化，增强了对讽刺样本的识别能力。

技术框架：整体架构包括双向门控交互模块和动态融合门，前者用于跨模态特征过滤，后者平衡模态重要性，生成更鲁棒的多模态表示。

关键创新：引入了动态门控跨模态融合和讽刺感知对比正则化，前者允许实例级别的模态贡献调整，后者则通过对比正则化增强语义一致性，显著提升了检测效果。

关键设计：采用多目标学习策略进行端到端训练，损失函数包括多模态分类损失和辅助单模态监督损失，确保模型在多模态和单模态任务上均表现良好。

🖼️ 关键图片

fig_0
fig_1
fig_2

📊 实验亮点

在MMSD和MMSD2.0数据集上的实验结果显示，所提方法在多模态讽刺检测任务中显著优于现有强基线，具体性能提升幅度达到XX%，验证了动态门控机制和对比正则化的有效性。

🎯 应用场景

该研究的潜在应用领域包括社交媒体内容分析、在线评论监测和人机交互系统等。通过提高讽刺检测的准确性，可以帮助系统更好地理解用户意图，从而提升用户体验和交互质量。未来，该技术可能在情感分析和舆情监测等领域发挥重要作用。

📄 摘要（原文）

Multimodal sarcasm detection aims to identify sarcastic intent from multimodal content, where inconsistencies between literal meaning and contextual cues often signal irony. This task has attracted increasing research attention. However, accurate detection remains challenging due to instance-dependent modality contributions and misleading semantic consistency, where surface-level alignment masks underlying contradictory intent. Existing methods often rely on fixed fusion strategies and treat sarcasm as generic cross-modal mismatch, limiting their ability to capture subtle sarcasm cues and instance-specific modality interactions. To address these challenges, we propose a novel MSD framework that integrates Dynamic Gated Cross-Modal Fusion with Sarcastic-aware Contrastive Regularization (SaCR). Specifically, a bidirectional gated interaction module performs cross-modal feature filtering and adaptively calibrates textual and visual contributions at the instance level. A dynamic fusion gate further balances modality importance to generate more robust multimodal representations. Furthermore, SaCR is introduced as a label-aware contrastive regularization objective that encourages semantic consistency for non-sarcastic samples while suppressing misleading consistency in sarcastic cases. The proposed framework is trained end-to-end with a multi-objective learning strategy that jointly optimizes multimodal classification and auxiliary unimodal supervision. Extensive experiments on MMSD and MMSD2.0 demonstrate that the proposed method consistently outperforms strong baselines.