VEglue: Testing Visual Entailment Systems via Object-Aligned Joint Erasing

📄 arXiv: 2403.02581v1 📥 PDF

作者: Zhiyuan Chang, Mingyang Li, Junjie Wang, Cheng Li, Qing Wang

分类: cs.CV, cs.SE

发布日期: 2024-03-05

备注: 12pages, 3 figures

💡 一句话要点

提出VEglue以解决视觉蕴含系统测试中的挑战

🎯 匹配领域: 支柱九：具身大模型 (Embodied Foundation Models)

关键词: 视觉蕴含 多模态推理 对象对齐 联合擦除 变形测试

📋 核心要点

  1. 现有的视觉蕴含系统测试方法多只考虑单一模态的扰动，导致无法有效检测图像与文本之间的关系。
  2. 本文提出VEglue，通过对齐图像与句子中的对象，设计联合擦除策略来增强测试的有效性。
  3. 实验结果显示，VEglue在问题发现率和检测问题数量上均显著优于现有基线，且能有效提升模型性能。

📝 摘要（中文）

视觉蕴含（VE）是一种多模态推理任务，涉及图像与句子对，旨在预测图像是否在语义上蕴含句子。现有的测试方法在处理VE系统时面临挑战，常常只考虑单一模态的扰动，导致测试效果不佳。为此，本文提出VEglue，一种基于对象对齐的联合擦除方法，通过对齐前提中的对象区域与假设中的对象描述，设计三种变形关系来联合擦除两个模态的对象。实验结果表明，VEglue在四个广泛使用的VE系统上检测到的平均问题数量为11,609个，较基线提升194%-2,846%。此外，VEglue的平均问题发现率为52.5%，显著优于基线，且在重新训练后模型性能提升50.8%。

🔬 方法详解

问题定义：本文旨在解决视觉蕴含系统测试中的有效性问题。现有方法往往只关注单一模态的扰动，无法有效捕捉图像与文本之间的语义关系，导致测试结果不可靠。

核心思路：VEglue的核心思路是通过对齐图像中的对象区域与句子中的对象描述，识别出相关和不相关的对象，从而设计出联合擦除的变形关系，以增强测试的深度和有效性。

技术框架：VEglue的整体架构包括对象对齐模块、变形关系设计模块和联合擦除模块。首先，通过对齐算法识别图像与文本中的对象，然后基于对齐信息设计变形关系，最后实施联合擦除操作。

关键创新：VEglue的最大创新在于其对象对齐的联合擦除方法，突破了传统方法仅考虑单一模态扰动的局限，能够更全面地评估视觉蕴含系统的性能。

关键设计：在设计过程中，VEglue采用了特定的对齐算法来识别对象，并设计了三种变形关系以实现联合擦除。此外，测试生成后用于重新训练模型，显著提高了模型在新测试上的准确性。

🖼️ 关键图片

fig_0
fig_1
fig_2

📊 实验亮点

实验结果显示，VEglue在四个视觉蕴含系统上平均检测到11,609个问题，较基线提升194%-2,846%。此外，VEglue的平均问题发现率达到52.5%，在重新训练后模型准确性提升50.8%，显著优于现有方法。

🎯 应用场景

VEglue的研究成果可广泛应用于视觉蕴含系统的测试与评估，尤其是在需要高准确性和可靠性的多模态推理任务中。其方法可以帮助开发者更好地理解和改进VE系统的性能，推动相关领域的研究与应用发展。

📄 摘要（原文）

Visual entailment (VE) is a multimodal reasoning task consisting of image-sentence pairs whereby a promise is defined by an image, and a hypothesis is described by a sentence. The goal is to predict whether the image semantically entails the sentence. VE systems have been widely adopted in many downstream tasks. Metamorphic testing is the commonest technique for AI algorithms, but it poses a significant challenge for VE testing. They either only consider perturbations on single modality which would result in ineffective tests due to the destruction of the relationship of image-text pair, or just conduct shallow perturbations on the inputs which can hardly detect the decision error made by VE systems. Motivated by the fact that objects in the image are the fundamental element for reasoning, we propose VEglue, an object-aligned joint erasing approach for VE systems testing. It first aligns the object regions in the premise and object descriptions in the hypothesis to identify linked and un-linked objects. Then, based on the alignment information, three Metamorphic Relations are designed to jointly erase the objects of the two modalities. We evaluate VEglue on four widely-used VE systems involving two public datasets. Results show that VEglue could detect 11,609 issues on average, which is 194%-2,846% more than the baselines. In addition, VEglue could reach 52.5% Issue Finding Rate (IFR) on average, and significantly outperform the baselines by 17.1%-38.2%. Furthermore, we leverage the tests generated by VEglue to retrain the VE systems, which largely improves model performance (50.8% increase in accuracy) on newly generated tests without sacrificing the accuracy on the original test set.