Pre-Flight: A Benchmark for Evaluating Large Language Models on Aviation Operational Knowledge
作者: Alex Brooker, Tim Hughes
分类: cs.AI, cs.CL
发布日期: 2026-07-02
备注: 9 pages, 1 figure, 2 tables. Benchmark available in inspect_evals (UKGovernmentBEIS/inspect_evals)
💡 一句话要点
提出Pre-Flight基准以评估航空领域大型语言模型的操作知识
🎯 匹配领域: 支柱九：具身大模型 (Embodied Foundation Models)
关键词: 航空操作知识 大型语言模型 评估基准 安全性评估 生成式AI 多项选择题 Inspect评估框架
📋 核心要点
- 现有的通用基准无法有效评估大型语言模型在航空特定操作知识上的推理能力，导致安全隐患。
- 本文提出Pre-Flight基准，通过300道多项选择题评估模型在航空操作知识上的表现，填补了这一空白。
- 实验结果表明，当前最强模型的准确率为82.7%，与专家水平存在显著差距，显示出该领域评估的必要性。
📝 摘要（中文）
大型语言模型（LLMs）在航空业务操作中的应用日益增多，但现有的通用基准无法有效评估模型在航空特定操作知识上的推理能力。为此，本文提出了Pre-Flight，一个开放源代码的基准，包含300道多项选择题，涵盖国际机场地面操作、ICAO和美国FAA法规、航空常识及复杂操作场景。这些问题由具备空中交通管理、地面操作和商业飞行经验的从业者撰写和审核。通过Inspect评估框架对多种商业和开放模型进行评估，结果显示即使是最强模型的准确率也仅为82.7%，远低于专家水平的95%。
🔬 方法详解
问题定义：本文旨在解决大型语言模型在航空操作知识推理中的评估不足，现有方法无法满足航空领域的高安全性要求。
核心思路：通过构建Pre-Flight基准，提供针对航空操作知识的专门评估工具，以确保模型在实际应用中的安全性和可靠性。
技术框架：该基准包含300道多项选择题，涵盖国际标准和机场地面操作材料，评估流程采用Inspect评估框架，依据标准的多项选择协议进行评分。
关键创新：Pre-Flight基准的提出是一个重要创新，它专注于航空领域的特定知识评估，与通用基准相比，提供了更具针对性的评估标准。
关键设计：问题由经验丰富的从业者撰写和审核，确保内容的专业性和准确性，评估结果通过准确率进行量化，并在新模型发布时持续更新排行榜。
🖼️ 关键图片
📊 实验亮点
实验结果显示，当前最强模型在Pre-Flight基准上的准确率为82.7%，相比于专家水平的95%仍有显著差距，表明在航空操作知识的推理能力上仍需进一步提升。这一发现强调了针对特定领域的评估的重要性。
🎯 应用场景
该研究的潜在应用领域包括航空公司、机场管理和航空培训机构等，能够帮助这些组织在使用生成式AI时确保模型的安全性和可靠性。未来，Pre-Flight基准可能成为航空领域AI应用的标准评估工具，推动行业的智能化发展。
📄 摘要（原文）
Large language models (LLMs) are increasingly proposed for aviation business operations, from documentation and training generation to customer facing assistants. General purpose benchmarks do not measure whether a model reasons safely and correctly about aviation specific operational knowledge, and the high stakes, regulated nature of the domain makes that gap consequential. We present Pre-Flight, an open source benchmark of 300 multiple choice questions drawn from international standards and airport ground operations material, covering international airport ground operations, ICAO and US FAA regulations, aviation general knowledge and complex operational scenarios. Questions were authored and reviewed by practitioners with experience in air traffic management, ground operations and commercial flying. We evaluate a range of contemporary commercial and open weight models using the Inspect evaluation framework, scoring by accuracy under a standard multiple choice protocol, and we maintain the leaderboard on a rolling basis as new models are released. Against an informal expert reference of around 95%, obtained from a low sample quiz of aviation professionals at a conference, even the strongest model evaluated (released in 2026) reaches 82.7%, having improved only gradually from roughly 75% in early 2025. A substantial and persistent gap below expert level reliability therefore remains. We release the dataset, the evaluation harness and the results, and the benchmark is available within the community evaluations package distributed with inspect_evals. We argue that domain specific evaluation of this kind is a necessary precondition for responsible deployment of generative AI in non safety critical aviation operations.