Paying More Attention to Visual Tokens in Self-Evolving Large Multimodal Models
作者: Shravan Venkatraman, Ritesh Thawkar, Omkar Thawakar, Rao Muhammad Anwer, Hisham Cholakkal, Salman Khan, Fahad Khan
分类: cs.CV
发布日期: 2026-06-25
备注: ECCV 2026
🔗 代码/项目: PROJECT_PAGE
💡 一句话要点
提出VISE框架以解决视觉内容关注不足问题
🎯 匹配领域: 支柱九：具身大模型 (Embodied Foundation Models)
关键词: 自我演化模型 多模态学习 视觉推理 无监督学习 几何不变性 语义不变性 视觉内容关注 模型优化
📋 核心要点
- 现有自我演化的多模态模型在视觉内容关注方面存在不足，导致生成结果依赖语言先验而非图像信息。
- 本文提出VISE框架，通过几何和语义不变性奖励直接优化模型的视觉调节策略，增强视觉内容的关注。
- 在18个基准测试中，VISE在COCO和TextCaps上分别提升了16.85和19.66的CIDEr分数，减少了对象幻觉现象。
📝 摘要（中文）
近年来，自我演化的大型多模态模型（LMMs）在纯无监督环境中提升视觉推理能力受到关注。然而，现有模型在优化答案一致性时未能确保解码器关注视觉内容，导致视觉欠调节现象，影响图像描述和视觉问答等任务的表现。为了解决这一问题，本文提出了VISE（视觉不变性自我演化）框架，通过几何不变性奖励和语义不变性奖励直接正则化模型的视觉调节策略。实验结果表明，VISE在多个基准测试中表现优异，显著提升了模型的性能。
🔬 方法详解
问题定义：本文旨在解决现有自我演化多模态模型在视觉内容关注不足的问题。现有方法在优化答案一致性时，解码器往往依赖语言先验，导致视觉信息的忽视。
核心思路：VISE框架通过引入几何不变性奖励和语义不变性奖励，直接正则化模型的视觉调节策略，确保解码器在生成过程中充分关注视觉内容。
技术框架：VISE框架在单一模型中运行，无需专业角色、外部奖励模型或标注，直接在原始未标记图像上进行训练。主要模块包括视觉调节策略和不变性奖励计算。
关键创新：VISE的核心创新在于引入两种不变性奖励机制，分别针对空间一致性和语义一致性进行优化，这与现有方法的依赖语言先验的做法形成鲜明对比。
关键设计：在设计中，几何不变性奖励确保模型在已知变换下保持空间一致性，语义不变性奖励则通过惩罚证据无关的生成来促使模型识别缺失证据的情况。
🖼️ 关键图片
📊 实验亮点
实验结果显示，使用Qwen3-VL-2B作为基础模型，VISE在COCO数据集上提升了16.85的CIDEr分数，在TextCaps上提升了19.66的CIDEr分数，同时减少了5.0 Chair-I点的对象幻觉现象，展现出良好的跨模型泛化能力。
🎯 应用场景
该研究的潜在应用领域包括图像描述生成、视觉问答以及其他需要视觉与语言理解的任务。通过提升模型对视觉内容的关注，VISE框架能够在多模态交互中提供更准确和一致的结果，具有重要的实际价值和未来影响。
📄 摘要（原文）
Recently, self-evolving large multimodal models (LMMs) have received attention for improving visual reasoning in a purely unsupervised setting. However, multi-role self-play and self-consistency reward schemes in existing self-evolving LMMs optimize answer agreement without ensuring the decoder attends to visual content, relying instead on statistical language priors to produce self consistent outputs. This leads to a persistent failure mode we term visual under-conditioning, where the decoder relies on language priors rather than the image during generation, manifesting as insufficient attention to visual tokens. As a result, current self-evolving LMMs struggle on vision--language understanding tasks such as image captioning and visual question answering. To address this, we propose VISE (Visual Invariance Self-Evolution), a purely unsupervised self-evolving framework that directly regularizes the model's visual conditioning policy through two complementary invariance-based rewards: a geometric invariance reward that enforces spatial consistency under known transformations, and a semantic invariance reward that penalizes evidence-agnostic generation by requiring the model to recognize the absence of evidence when predicted regions are perturbed. VISE operates within a single model without specialist roles, external reward models, or annotations, and is trained on raw unlabeled images. Experiments on 18 benchmarks demonstrate the efficacy of our approach. Using Qwen3-VL-2B as the base model, VISE achieves gains of $+16.85$ CIDEr on COCO and $+19.66$ CIDEr on TextCaps, reduces object hallucination by $5.0$ Chair-I points, and generalizes across four model families and scales. Our code and models are available at https://mbzuai-oryx.github.io/VISE