WorldCup Arena: Prospective, Leakage-Free Evaluation of Frontier LLMs on a Live Tournament
作者: Zhenran Wang, Zhonghan Bian, Jinsong Li, Zhangyang Qi
分类: cs.CL
发布日期: 2026-08-04
备注: Project page: https://co-minder.github.io/worldcup2026
💡 一句话要点
提出一种无泄漏的前沿LLM评估方法以预测世界杯赛事
🎯 匹配领域: 支柱九：具身大模型 (Embodied Foundation Models)
关键词: 大型语言模型 世界杯预测 无泄漏评估 实时预测 模型评估
📋 核心要点
- 现有的LLM评估方法多为回顾性，容易受到记忆化影响，缺乏前瞻性和真实预测能力的测试。
- 本文提出在世界杯期间实时评估LLM的预测能力，确保在问题提出时没有答案可供参考，从而实现无泄漏评估。
- 实验结果显示，六个LLM在比赛结果预测上的准确率为63.9%，但它们的共识性高于准确性，且在某些预测上表现不足。
📝 摘要（中文）
现有的大型语言模型（LLM）评估通常是回顾性的，存在记忆化的风险。本文设计了一种前瞻性的评估方法，在2026年世界杯期间，六个前沿LLM在每场比赛开始前被要求填写预测卡，确保评估过程无泄漏。结果显示，这些模型在比赛结果上的平均准确率为63.9%，与博彩公司偏好相当，但它们之间的共识往往高于正确率，且在平局和进球预测上表现不足。研究还揭示了当前前沿系统在预测能力上的差异不大，整体表现保持稳定。
🔬 方法详解
问题定义：本文旨在解决现有LLM评估方法的回顾性问题，避免记忆化带来的偏差，提供一种真实的预测能力测试。
核心思路：通过在2026年世界杯期间，要求LLM在比赛开始前进行实时预测，确保每个问题在提出时没有现成答案，从而实现无泄漏评估。
技术框架：整体流程包括比赛前的预测请求、LLM的实时预测生成、结果记录与评分，形成一个完整的评估体系。主要模块包括预测生成模块和结果评分模块。
关键创新：本研究的创新在于设计了一种无泄漏的评估机制，通过实时预测而非回顾性评估，确保了评估的真实性和有效性。
关键设计：在模型的参数设置上，确保了LLM能够进行深度推理和实时搜索，使用了适合的损失函数来优化预测准确性。
🖼️ 关键图片
📊 实验亮点
实验结果显示，六个前沿LLM在比赛结果预测上的平均准确率为63.9%，与博彩公司偏好相当。尽管它们之间的共识性较高，但在平局和进球预测上表现不足，揭示了当前模型在某些情况下的局限性。
🎯 应用场景
该研究的潜在应用领域包括体育赛事预测、金融市场分析和其他需要实时决策的场景。通过无泄漏的评估方法，可以更准确地评估模型的预测能力，推动相关领域的研究与应用发展。
📄 摘要（原文）
Benchmarks that measure the forecasting ability of large language models are almost always retrospective: the event has happened, the answer is somewhere on the Web, and the evaluation must defend itself against memorisation. We report the opposite design. Over the 39 days of the 2026 FIFA World Cup, six frontier LLMs -- all with extended thinking and native server-side web search -- were asked before every kickoff, one match at a time, to fill in a seven-market prediction card for all 104 matches, plus 12 group winners and a pre-tournament outright pool; no answer existed when the question was asked, so the evaluation is leakage-free by construction rather than by filtering, and the frozen archive holds 4,494 scored predictions. What the tournament establishes is a set of behaviours the six systems share. On match outcome they average 63.9%, level with backing the bookmaker's favourite -- which is in fact what they usually do. They agree with one another far more often than they are right, so a majority vote adds nothing. They under-commit to draws and to goals, and crowd their scoreline picks onto a single prototypical result. Accuracy tracks how lopsided a fixture is rather than how much is known about it: it collapses in the closest ties, where the dossiers are richest, while questions about the tournament as a whole are answered well. On this task the current generation of frontier systems is not sharply differentiated: the standings hold up at the top and the bottom across the run and churn in the middle, and the margins stay narrow throughout. The briefing dossiers, fixtures and official results are released as a benchmark, together with the scoring code.