CaliBench: Are the Stochastic Dynamics of Video World Models Physically Calibrated?
作者: Jonathan Sadeghi, Jenny Seidenschwarz, Jesse Allardice, Sirish Srinivasan, Benjamin Graham, Jeffrey Hawke
分类: cs.LG, cs.AI
发布日期: 2026-08-17
💡 一句话要点
提出CaliBench以解决视频世界模型的物理校准问题
🎯 匹配领域: 支柱二：RL算法与架构 (RL & Architecture)
关键词: 视频生成 物理校准 随机不确定性 模型评估 卡方检验 离散空间评分 生成模型
📋 核心要点
- 现有视频世界模型的基准测试未能有效评估特定现象的细粒度随机不确定性，导致模型校准性不足。
- CaliBench通过在物理可解释的离散空间中评分，直接测量与已知参考分布的距离，从而提供精确的校准测试。
- 实验结果表明，大多数模型在多个场景中显著失配，且模型在不同场景中的表现差异明显，未能普遍优越。
📝 摘要（中文）
视频世界模型通过生成采样来近似物理结果的随机分布，但现有基准仅对单个生成结果进行评分或粗略比较整个数据集的分布，未能测试特定现象的细粒度随机不确定性。我们提出CaliBench，它在物理可解释的离散空间中评分结果，而非学习特征空间，从而直接测量与已知参考分布的距离。我们策划了已知参考的结果空间，进行精确的校准测试，并将性能分解为可评分性和校准度两个正交轴。通过卡方检验评估显著性，结果显示大多数场景模型组合显著失配，且模型在不同场景中的表现差异明显。我们发布了该协议及比较新模型的指标。
🔬 方法详解
问题定义：论文旨在解决现有视频世界模型在物理校准方面的不足，现有方法仅对生成结果进行粗略评分，未能有效捕捉细粒度的随机不确定性。
核心思路：CaliBench通过在物理可解释的离散空间中评分结果，直接与已知的参考分布进行比较，从而实现精确的校准测试。
技术框架：整体架构包括结果空间的策划、评分机制的设计和性能评估三个主要模块。结果空间基于已知的参考分布，评分机制则通过可评分性和校准度两个维度进行评估。
关键创新：CaliBench的主要创新在于其评分机制的设计，能够在物理可解释的离散空间中直接测量与参考分布的距离，区别于传统方法的学习特征空间评分。
关键设计：在实验中，采用卡方检验来评估模型的校准性，设置了32次生成以确保结果的可靠性，且引入了均值归一化总变差（mnTV）作为比较新模型的指标。
🖼️ 关键图片
📊 实验亮点
实验结果显示，大多数模型在九个场景中显著失配，尤其是Veo 3.1在骰子生成中几乎崩溃至单一结果。模型在轮盘生成中表现出低可评分性，整体表现因场景而异，未有模型在所有场景中占优。
🎯 应用场景
CaliBench的研究成果可广泛应用于视频生成、物理模拟和机器学习模型的评估等领域。其精确的校准测试方法能够帮助研究人员更好地理解和改进生成模型的性能，推动相关技术的进步与应用。
📄 摘要（原文）
Video world models approximate the stochastic distribution of physical outcomes through generative sampling, but existing benchmarks score individual generations or compare distributions coarsely over a whole dataset, leaving the fine-grained aleatoric uncertainty of specific phenomena untested. We introduce CaliBench, which scores outcomes in a physically interpretable discrete space - a bin index, a die face, a suit, a colour - rather than a learned feature space such as in FID, so the distance from a known reference distribution is measured directly. We curate outcome spaces whose reference is known in closed form (binomial Galton boards, Bernoulli forks, uniform dice/cards/lottery, a skewed European-roulette colour), enabling an exact calibration test. We decompose performance into two orthogonal axes that a single accuracy metric conflates: scorability, the fraction of generations yielding a scoreable outcome, and calibration, the total variation distance from the reference on that sample. A chi-squared test assesses significance; as calibration is its null hypothesis it can evidence only miscalibration, and at N=32 per cell detects only large deviations. We apply it to nine scenes and six image-to-video models (WAN-2.7, SeeDance-2.0, HappyHorse-1.0, Veo 3.1, Runway Gen-4.5, Cosmos3-Super), 32 generations each. Models consistently concentrate probability mass on a few outcomes rather than reproducing the reference. Most scene-model combinations are significantly miscalibrated, in the extreme collapsing to one outcome, as Veo 3.1 does on dice. On roulette, generations often leave the ball ambiguously placed, giving several models low scorability. Performance varies by scene: no model dominates all nine. We release the protocol and a metric (mean normalised total variation, mnTV) for comparing new models against our results.