OmniJudge or OmniBias? Diagnosing Multimodal Judges through Balanced, Decoupled Lenses
作者: Guangzheng Hu, Ziyue Jiang, Weixu Qiao, Lixin Zhang, Jianye Kang, Yuru Wu, Rong Bao, Niantong Li, Wei Wang, Ziyi Cheng, Xinfa Zhu, HangRui Hu, Ting He, Bing Zhao, Lin Qu, Hu Wei, Jin Xu
分类: cs.AI
发布日期: 2026-08-25
💡 一句话要点
提出D3-Omni基准以解决多模态评估中的偏差问题
🎯 匹配领域: 支柱九：具身大模型 (Embodied Foundation Models)
关键词: 多模态理解 评估基准 系统性盲点 文本到图像 文本到视频 文本到语音 数据平衡 解耦评估
📋 核心要点
- 现有多模态理解模型在评估时存在偏差，无法准确识别不同类型的失败模式。
- 本文提出D3-Omni基准，通过固定正例种子并使用控制性提示重写和扰动生成负例，解决评估中的不平衡问题。
- 实验结果表明，强大的OmniJudges在处理模态相关维度时表现不佳，且往往将不同属性视为单一决策，揭示了系统性盲点。
📝 摘要（中文）
多模态理解模型在文本到图像、视频和语音生成的评估中被称为“OmniJudges”，但其评分的可靠性尚不明确。现有基准和训练数据往往过于强调正例，混淆不同的失败模式，导致评估者可能在未识别失败的情况下获得高分。为此，本文提出了D3-Omni，一个平衡且解耦的基准，涵盖53个正交二元维度和10,671个样本。D3设计具有双重平衡、解耦和动态特性，旨在揭示和解决系统性盲点。
🔬 方法详解
问题定义：本文旨在解决多模态理解模型在评估时的偏差问题，现有方法往往无法准确识别不同的失败模式，导致评估结果不可靠。
核心思路：论文提出D3-Omni基准，通过平衡和解耦的方式，确保每个错误都能归因于单一能力，进而揭示模型的系统性盲点。
技术框架：D3-Omni基准包含53个正交二元维度，分为文本到图像、视频和语音生成三个任务，采用固定正例种子和控制性扰动生成负例，确保数据的多样性和代表性。
关键创新：D3-Omni的双重平衡、解耦和动态特性是其主要创新，与现有方法相比，能够更好地揭示模型的能力差距和盲点。
关键设计：在设计中，采用了固定的正例种子，结合控制性提示重写和维度隔离扰动，确保每个维度的样本均衡，且通过动态调整引导生成模型向未充分代表的标签分布区域发展。
🖼️ 关键图片
📊 实验亮点
实验结果显示，D3-Omni基准使得每个维度的样本达到近1:1的平衡，并在总分级别上实现均匀分布。强大的OmniJudges在模态相关维度上表现不佳，揭示了其在识别违反要求方面的系统性盲点。
🎯 应用场景
该研究的潜在应用领域包括多模态内容生成的自动评估、智能推荐系统以及人机交互等。通过提供更可靠的评估工具，D3-Omni可以帮助提升多模态模型的性能，推动相关技术的实际应用和发展。
📄 摘要（原文）
Multimodal understanding models that can jointly judge text-to-image (T2I), text-to-video (T2V) and text-to-speech (TTS) generation are increasingly used as "OmniJudges" for evaluation and automatic annotation. How reliably they understand what they score remains unclear, since existing benchmarks and training data tend to overemphasize positive examples and to conflate distinct failure modes, so a judge may score well without recognizing failures while its capability gaps stay hidden. Motivated by this, we introduce D3-Omni, a balanced and decoupled benchmark for diagnosing fine-grained multimodal understanding, covering 53 orthogonal binary dimensions (17/22/14) and 10,671 samples (3,526/1,998/5,147) across the three tasks. Rather than re-generating outputs, which may leak information across dimensions, we fix verified fully positive seeds and derive negatives through controlled prompt rewriting and atomic, dimension-isolating perturbations. The resulting D3 design is Dual-balanced, which helps alleviate negative-sample scarcity and per-dimension label imbalance; Decoupled, so that each error is attributable to a single capability; and Dynamic, steering construction toward under-represented regions of the label distribution as generative models improve.The suite reaches near 1:1 per-dimension parity and a uniform distribution over all total-score levels. Under this balanced view, even strong OmniJudges tend to struggle on modality-related dimensions, to confirm satisfied requirements far more reliably than they detect violated ones, and to treat nominally distinct attributes as largely a single decision, suggesting that aggregate accuracy may hide systematic blind spots that a balanced and decoupled lens can help expose and, in turn, address.