PRISM-BENCH: Diagnosing Grounded and Faithful Multimodal Reasoning Beyond Final-Answer Accuracy

📄 arXiv: 2610.36554v1 📥 PDF

作者: Vansh Bhatia, Mehdi Rezagholizadeh, Parsa Ashrafi Fashi, Igor Gilitschenski, Vikram Appia, Emad Barsoum

分类: cs.CV, cs.CL

发布日期: 2026-10-27

备注: Accepted as a poster at the NeurIPS 2026 Workshop on Grounded and Faithful Vision-Language Models for Real-World Deployment (VLM4RWD). 18 pages, 10 figures, 7 tables. Dataset: https://huggingface.co/datasets/vanshbhatia/PRISM-BENCH

💡 一句话要点

提出PRISM-BENCH以解决多模态推理的准确性不足问题

🎯 匹配领域: 支柱九：具身大模型 (Embodied Foundation Models)

关键词: 多模态推理 视觉-语言模型 模型评估 推理轨迹 CORE指标

📋 核心要点

  1. 现有视觉-语言模型的评估方法主要关注最终答案的准确性，忽视了推理过程的有效性和答案的生成成本。
  2. 本文提出PRISM-BENCH基准，通过专家策划的问题和理想化推理轨迹，提供了更全面的模型评估方式。
  3. 实验结果表明，轨迹对齐能够有效预测答案的正确性，并且CORE指标在准确性和响应长度之间保持无关性。

📝 摘要（中文）

现有的视觉-语言模型评估主要依赖最终答案的准确性，但未能考虑答案是否与图像相结合、推理过程是否反映证据以及答案生成的成本。为此，本文提出了PRISM-BENCH，一个包含1000个专家策划问题的基准，涵盖五类少数代表性推理，每个问题都有理想化的推理轨迹。通过感知探测和轨迹对齐，本文揭示了模型失败的主要原因，并引入了CORE指标以提高评估效率。实验结果显示，最佳系统的准确率为66.0%，但在与人类基线的比较中仍有18.4分的差距。

🔬 方法详解

问题定义：本文旨在解决现有视觉-语言模型评估中仅依赖最终答案准确性的问题，缺乏对推理过程和答案生成成本的考量。

核心思路：通过构建PRISM-BENCH基准，提供专家策划的问题和理想化的推理轨迹，以便更全面地评估模型的推理能力和准确性。

技术框架：PRISM-BENCH包含1000个问题，分为五类推理，每个问题都有由人工标注的推理轨迹。模型评估包括感知探测和轨迹对齐两个主要模块。

关键创新：引入了CORE指标，能够在准确性和响应长度之间保持无关性，并重新排序模型对以提高评估效率。

关键设计：在实验中，使用了25个开放和封闭系统，包括GPT-5和Claude Opus 4.6，评估结果显示最佳准确率为66.0%。

🖼️ 关键图片

fig_0
fig_1
fig_2

📊 实验亮点

实验结果显示，轨迹对齐方法能够有效预测25个系统的答案正确性，宏观AUROC达到0.66。同时，CORE指标在准确性上重新排序了24.2%的模型对，显示出其在评估效率上的优势。

🎯 应用场景

该研究的潜在应用领域包括智能问答系统、图像理解和多模态交互等。通过提供更全面的评估标准，PRISM-BENCH能够帮助开发更可靠的视觉-语言模型，推动相关技术在实际场景中的应用和发展。

📄 摘要（原文）

Final-answer accuracy reports whether a vision-language model was right, but not whether its answer was grounded in the image, whether its stated reasoning reflects that evidence, or what the answer cost to produce. Those are the properties that decide whether a system can be deployed. We present PRISM-BENCH, a benchmark of 1,000 expert-curated questions spanning five under-represented reasoning categories, in which every item carries an idealized reasoning trace written by one human annotator and validated by another. The traces support two diagnostics that accuracy cannot provide. A perception probe, in which the model answers from a description it wrote itself, attributes roughly 85% of frontier-model failures to misperception and places the bottleneck at grounding. Trace alignment predicts answer correctness for all 25 systems tested (macro AUROC 0.66) and survives length control, so it measures reasoning content and not verbosity. We further introduce CORE, an efficiency-aware metric that reorders 24.2% of model pairs relative to accuracy while remaining uncorrelated with response length. Across 25 open and closed systems, including GPT-5 and Claude Opus 4.6, the best accuracy is 66.0%. On the four categories with a human baseline the best system trails humans by 18.4 points, and recent frontier gains fall on recognition and not on spatial grounding.