Same Outputs, Different Scores: Auditing the Unchecked Graders of Multimodal LLM Benchmarks

📄 arXiv: 2611.03377v1 📥 PDF

作者: Ahmad Jarjis Hasan, Jonathan Lwowski

分类: cs.AI, cs.CL, cs.CV

发布日期: 2026-11-02

备注: Accepted at WACV 2027 (Evaluations and Datasets Track). Code and data: https://github.com/ahmadhasan2k8/grader-audit

💡 一句话要点

提出审计多模态LLM基准评分规则以解决评分不一致问题

🎯 匹配领域: 支柱九：具身大模型 (Embodied Foundation Models)

关键词: 多模态学习 评分机制 模型评估 人工智能基准 盲评方法

📋 核心要点

  1. 现有多模态LLM基准的评分规则缺乏透明性和合理性，导致评分结果可能不准确。
  2. 本文提出了一种盲人类与LLM结合的评分方法，能够更准确地反映模型的实际表现。
  3. 实验结果显示，评分规则的不同可以导致高达22分的准确率差异，强调了评分标准的重要性。

📝 摘要（中文）

每个多模态LLM的多项选择基准都依赖于评分规则，将模型的响应转化为对错判定，而这一规则通常未经检验。本文重现了八个同行评审基准的官方评分，并引入了盲人类与LLM评判的结合方法，发现评分规则对相同输出的准确率影响可达22分。研究表明，现有评分标准不可靠，建议基准报告不合规率并使用至少两种评分规则。

🔬 方法详解

问题定义：本文旨在解决多模态LLM基准中评分规则的不透明性和不一致性问题。现有方法的痛点在于评分规则未经过检验，可能导致模型表现的误判。

核心思路：论文提出通过盲人类与LLM结合的评分方法，来更准确地评估模型的输出，避免单一评分规则带来的偏差。

技术框架：整体架构包括数据集的重现、评分规则的分析、盲评机制的设计等多个模块。首先重现八个基准的评分规则，然后引入人类与LLM的结合评分机制。

关键创新：最重要的技术创新在于引入了盲人类与LLM的联合评分方法，显著提高了评分的准确性和可靠性，解决了传统评分方法的局限性。

关键设计：在评分过程中，采用了Cohen's kappa来评估评分一致性，确保评分的客观性和准确性。同时，针对不同输出形式设计了多种评分规则，以应对模型输出的多样性。

📊 实验亮点

实验结果显示，评分规则的不同可以导致高达22分的准确率差异，尤其在模型输出为自由文本时，评分规则的影响更为显著。此外，某些基准的非合规率高达零，表明现有评分机制的不足。

🎯 应用场景

该研究的潜在应用领域包括多模态机器学习基准的设计与评估、人工智能模型的性能评估等。通过提高评分的透明性和准确性，能够为研究人员和开发者提供更可靠的模型性能指标，推动多模态AI的发展。

📄 摘要（原文）

Every multimodal-LLM multiple-choice benchmark ends with a scoring rule that converts a model's response, often free-form text rather than a clean option letter, into a right/wrong verdict, buried in its code. Unlike the dataset, this rule is chosen without justification and left unexamined, yet it sets the published number. We reproduce the official scorers of eight peer-reviewed benchmarks on five open vision-language models (VLMs) that fit a single 16 GB GPU, and introduce a blinded human-plus-LLM-judge adjudication (Cohen's kappa=0.956) that recovers what the model actually answered. The scoring rule alone moves accuracy by up to 22 points on identical outputs, yet exactly 0 wherever a model emits clean option letters, so it hides on strong-model leaderboards. It is the fallback and not the rambling that does the damage: the benchmark with the highest non-compliance we measure has an artifact of exactly zero, because its scorer abstains instead of guessing; a reasoning model with non-compliance 0.000 still swings 9.5 points with which regex match of its trace the parser keeps. No standard rule is reliable: snapping a non-answer to its closest option credits answers never given yet buries correct ones, abstention discards answers stated in words, and likelihood scoring follows answer-token priors, so a single grade conflates capability with luck rather than measuring either. We release all code, labels, and the protocol, and argue benchmarks should report a non-compliance rate and at least two rules, not one unexamined grade.