SciCode-Verified: How Benchmark Defects Underestimated the Scientific-Coding Ability of Language Models
作者: Sihan Hu, Lyuhan Huang, Youjin Deng, Kun Chen
分类: cs.SE, cs.AI
发布日期: 2026-08-05
备注: 47 pages, 2 figures, 6 tables. Project repository: https://github.com/flyingwagner/scicode-verified
💡 一句话要点
提出SciCode-Verified以解决科学编码能力评估的缺陷问题
🎯 匹配领域: 支柱九：具身大模型 (Embodied Foundation Models)
关键词: 科学编码 语言模型 评估标准 模型能力 领域专家审计 性能提升
📋 核心要点
- 现有的SciCode评估方法存在缺陷，导致语言模型的科学编码能力被低估。
- 论文通过对65个测试问题进行专家审计，发现并修正了263个缺陷，提出了SciCode-Verified。
- 在修正后的基准上，前沿模型的子问题准确率和主问题准确率均显著提升，显示出模型的真实能力。
📝 摘要（中文）
SciCode是评估语言模型科学编码能力的标准，但其得分近期停滞不前。研究发现，65个测试问题中存在263个缺陷，导致正确的解决方案被错误拒绝。通过对这些缺陷的修正，论文提出了SciCode-Verified，并重新评估了前沿模型，结果显示子问题准确率从45-60%提升至84-98%，主问题准确率从9-27%提升至69-92%。这表明，现有模型在科学编码方面的能力被低估，瓶颈在于评估工具的质量。
🔬 方法详解
问题定义：论文旨在解决SciCode评估语言模型科学编码能力时存在的缺陷，现有方法导致模型能力被低估，得分停滞不前。
核心思路：通过对所有测试问题进行领域专家审计，识别并修正影响评估结果的缺陷，从而提高评估的准确性和可靠性。
技术框架：整体流程包括问题审计、缺陷识别、修正实施和重新评估四个主要阶段。每个阶段都由领域专家参与，确保修正的有效性。
关键创新：最重要的创新在于通过系统的审计发现并修正了263个缺陷，特别是78%的缺陷需要专业知识才能识别，显著提升了评估的质量。
关键设计：在修正过程中，添加了必要的规范，修复了评分标准，收紧了过于宽松的测试要求，所有更改均有记录并经过第二位专家的独立检查。
🖼️ 关键图片
📊 实验亮点
在修正后的SciCode-Verified基准上，前沿模型的子问题准确率从45-60%提升至84-98%，主问题准确率从9-27%提升至69-92%。这一结果表明，现有模型在科学编码方面的能力远超之前的评估结果，显示出评估工具的质量对模型能力的影响。
🎯 应用场景
该研究的潜在应用领域包括科学计算、教育和科研评估等。通过提供更准确的模型评估标准，SciCode-Verified可以帮助研究人员更好地理解和利用语言模型在科学编码中的能力，推动相关领域的研究进展。
📄 摘要（原文）
SciCode is the standard measure of the scientific-coding ability of language models: research-level problems that demand both frontier scientific theory and its implementation as working numerical code. It is a component of the Artificial Analysis Intelligence Index and a standing evaluation in government and national-laboratory suites. Yet its scores have recently plateaued: the strongest 2026 models cluster tightly around 60\% subproblem accuracy, and a successor model ties its predecessor. We trace this stagnation to defects in the benchmark itself. A per-problem, domain-expert audit of all 65 test problems uncovers 263 defects; 192 of them, spread across 91\% of the main problems, cause correct, instruction-following solutions to be wrongly rejected---through non-reproducible gold answers, over-tight tolerances, or self-contradictory specifications. Critically, 78\% of these score-suppressing defects require specialized physics or mathematics knowledge to detect, not mere clerical proofreading. We corrected every confirmable defect to produce SciCode-Verified. The corrections add only the specifications a well-posed problem requires, repair grading, and tighten the tests that were too lenient; every change is recorded with its justification and independently re-checked by a second domain expert. We re-evaluate twelve frontier model snapshots on the corrected benchmark and find a substantial recovery: subproblem accuracy rises from 45--60\% to 84--98\%, and main-problem accuracy from 9--27\% to 69--92\%. State-of-the-art models are far more proficient in scientific coding than SciCode has suggested---the bottleneck was not model capability, but the quality of the evaluation instrument. We release SciCode-Verified with its complete audit trail as the corrected public standard.