AGC-Bench: Measuring Artificial General Creativity
作者: Roger Beaty, Vijeta Deshpande, Clin K.Y. Lai, Anna Attuch, Namrata Shivagunde, Swastik Roy, Rajkumar Pujari, Paul V. DiStefano, Sherin Muckatira, Claire E. Stevenson, Mikhail Gronas, Anna Rumshisky
分类: cs.CL
发布日期: 2026-07-05
💡 一句话要点
提出AGC-Bench以衡量人工通用创造力
🎯 匹配领域: 支柱六：视频提取与匹配 (Video Extraction)
关键词: 人工通用创造力 大型语言模型 评判者反应理论 创造力基准 数据集构建 心理测量 模型微调
📋 核心要点
- 现有的创造力评估缺乏统一的基准，尤其是在大型语言模型（LLMs）领域，难以衡量其创造力表现。
- 论文提出AGC-Bench，通过系统性文献回顾和评判者反应理论，构建了一个标准化的创造力评估框架。
- 实验结果表明，LLMs在不同领域的创造力表现存在差异，且顶尖人类创造力仍优于顶尖LLM。
📝 摘要（中文）
创造力研究一直在争论创造力是否特定于某一领域（如视觉、写作、科学），以及它是否在心理测量上与一般智力可分。针对这一问题，本文提出了AGC-Bench，一个基于系统性文献回顾构建的人工通用创造力基准。该基准涵盖了78个数据集，涉及头脑风暴、问题解决、STEM、叙事、比喻语言和幽默等领域。通过应用评判者反应理论，校准了评判者的宽松/严厉程度，并对Qwen3-30B进行了微调，生成了AGC-Judge，一个开放权重模型，能够稳健地评分未训练的新创造力基准。实验结果显示，前沿模型在AGC-Bench排行榜上表现优异，且开放模型紧随其后。
🔬 方法详解
问题定义：论文要解决的问题是如何统一衡量人工智能的创造力，现有方法缺乏标准化基准，导致评估结果不一致。
核心思路：论文的核心思路是通过AGC-Bench构建一个系统化的创造力评估框架，结合评判者反应理论来校准评判者的评分偏差，从而提高评估的准确性和可靠性。
技术框架：整体架构包括文献回顾、数据集构建、评判者反应理论应用和模型微调等主要模块。首先筛选出相关文献，识别出497个基准，然后构建78个数据集，最后通过AGC-Judge进行评分。
关键创新：最重要的技术创新点在于引入了评判者反应理论来校准评分偏差，并通过微调模型生成AGC-Judge，使其能够在未见过的基准上进行稳健评分。与现有方法相比，AGC-Bench提供了更为系统和标准化的评估手段。
关键设计：在模型微调过程中，采用了偏差校正的评分数据，并设计了开放权重的AGC-Judge模型，以确保其在不同创造力基准上的适应性和准确性。
🖼️ 关键图片
📊 实验亮点
实验结果显示，经过因子分析，发现一个单一的创造力因子'c'，解释了81.5%的方差，且在一些领域（如写作）LLMs的表现优于其他领域（如科学构思）。此外，顶尖人类创造力仍领先于顶尖LLM，显示出人类在创造力方面的独特优势。
🎯 应用场景
该研究的潜在应用领域包括教育、创意产业和人工智能开发等。通过AGC-Bench，研究人员和开发者可以更有效地评估和比较不同AI模型的创造力表现，从而推动创造性AI的进一步发展和应用。
📄 摘要（原文）
Creativity research has debated whether creativity is domain-specific (e.g., visual, writing, science), and if it is psychometrically separable from general intelligence. Both questions now apply to LLMs, but a unified benchmark of AI creativity remains elusive. We introduce AGC-Bench, an artificial general creativity benchmark built from a systematic review of the AI creativity literature (3,101 papers screened, 497 benchmarks identified), paired with an agentic harness that converts idiosyncratic codebases into HELM-standardized benchmarks. The first release covers 78 datasets spanning brainstorming, problem solving, STEM, narrative, figurative language, and humor. To address bias in LLM-as-judge, we apply Judge Response Theory -- a psychometric calibration of judge leniency/severity; we then fine-tune Qwen3-30B on the bias-corrected ratings of three frontier LLMs to produce AGC-Judge, an open-weight model that robustly scores new creativity benchmarks it was not trained on. Results reveal frontier models at the top of the AGC-Bench leaderboard, with open models close behind. LLMs show different creative strengths, ranking higher on some domains (e.g., writing) than others (e.g., scientific ideation). Extensive experiments yield three main findings. First, applying factor analysis across 83 LLMs, we recover a single creativity factor 'c', analogous to the 'g' factor of general intelligence, that explains 81.5% of variance, related to but separable from general knowledge/reasoning. Second, we show that prompting models to "be creative" boosts their performance far more than enabling reasoning, evidence that the benchmark tracks creativity over general ability. Third, on a human-matched subset, we find the top human still leads the top LLM on creativity. We release AGC-Bench with a public leaderboard, AGC-Judge, and human data as open infrastructure for measuring AI creativity at scale.