Patient-specific rubric-based evaluation of large language models across longitudinal cancer care
作者: Wei-Wei Zhang, Pengkai Wang, Yan Li, Zhijie Sang, Guan-Qun Zhou, Jun-Yan Li, Shu-Han Zhou, Pei-Wen Liu, Ting-Xi Tang, Min Tang, Zhitian Hou, Jianmin Wu, Jian Guan, i Lin, Hongxia Yang, Ying Sun
分类: cs.AI
发布日期: 2026-10-26
💡 一句话要点
提出NPC-RubricBench以评估癌症护理中的语言模型表现
🎯 匹配领域: 支柱九：具身大模型 (Embodied Foundation Models)
关键词: 语言模型 癌症护理 临床推理 多阶段评估 动态评估 决策支持 医疗AI
📋 核心要点
- 现有的语言模型评估方法主要关注单次临床接触，未能有效反映长期护理中的临床推理演变。
- 论文提出NPC-RubricBench基准，专注于癌症护理的多阶段评估，强调阶段适应性的重要性。
- 实验结果表明，医学适应模型的遗漏率高于通用模型，且开放式请求会降低决策质量。
📝 摘要（中文）
大型语言模型（LLMs）在专业医学中展现出潜力，但现有评估主要集中于孤立的临床接触，而未能跟踪临床责任在长期护理路径中的演变。本文引入NPC-RubricBench，这是一个针对癌症护理连续体中多阶段临床推理的评估基准，采用鼻咽癌作为具体疾病背景。该基准从405名患者的纵向记录中整理而成，包含941个患者阶段单元，涵盖六个连续护理阶段和17,688个患者特定的原子标准，评估严格基于决策时的临床证据。对30个通用和医学LLMs的评估显示，强大的诊断性能并未转化为下游护理的改善，且主导失败模式在各阶段之间发生逆转。
🔬 方法详解
问题定义：本文旨在解决现有语言模型在癌症护理中评估不足的问题，尤其是缺乏对临床责任演变的跟踪。现有方法未能反映多阶段护理中的复杂性和动态变化。
核心思路：论文提出NPC-RubricBench基准，通过对405名患者的纵向记录进行整理，构建一个涵盖多个护理阶段的评估框架，以便更全面地评估语言模型的临床推理能力。
技术框架：该框架包括941个患者阶段单元，分为六个护理阶段，涵盖17,688个原子标准，评估严格基于决策时的临床证据，确保评估的准确性和相关性。
关键创新：最重要的创新在于引入了阶段适应性评估，强调静态诊断基准不足以确保临床可靠性，提出了动态评估的必要性。
关键设计：在模型评估中，采用了多维度的评估标准，关注决策质量、证据可信度和安全遵循性等关键参数，设计了不同粒度的查询以测试模型的表现。
🖼️ 关键图片
📊 实验亮点
实验结果显示，医学适应模型的决策遗漏率高达33.4%，而通用模型的遗漏率相对较低。此外，开放式请求导致所有30个模型的决策质量下降，强调了查询粒度对模型表现的显著影响。
🎯 应用场景
该研究的潜在应用领域包括癌症护理、临床决策支持系统和医疗AI的开发。通过提供一个动态评估框架，能够帮助医疗机构更好地选择和应用语言模型，从而提升患者护理质量和安全性。未来，NPC-RubricBench可能推动更广泛的医学AI评估标准的建立。
📄 摘要（原文）
Large language models (LLMs) show promise in specialty medicine, yet existing evaluations largely focus on isolated clinical encounters rather than tracking how clinical responsibilities evolve across a longitudinal care pathway. Here we introduce NPC-RubricBench, a phase-aware benchmark for evaluating multistage clinical reasoning across the cancer care continuum, using nasopharyngeal carcinoma as a defined disease context. Curated from longitudinal records of 405 patients across two cancer centers, the benchmark comprises 941 patient-phase units across six sequential care phases and 17,688 patient-specific atomic criteria spanning five evaluation dimensions, with each assessment strictly restricted to decision-time clinical evidence. Evaluating 30 general-purpose and medical LLMs revealed that strong diagnostic performance does not carry over to downstream care. We evaluated 30 general-purpose and medical LLMs, showing that high diagnostic performance did not translate into improved downstream care. Instead, the dominant failure modes reversed across phases. Explicit decision errors dominated anatomical staging at 48.2%, whereas decision omissions dominated initial work-up and post-treatment follow-up at 33.4% and 30.4%, respectively. In the evaluated model set, medically adapted models showed higher omission rates than general-purpose models. Moreover, varying query granularity exposes a critical specification trade-off: broad open-ended requests degrade Decision Quality across all 30 models, while more detailed, checklist-like prompts raise Decision Quality but reduce Evidence Faithfulness and Safety Adherence. Furthermore, NPC-RubricBench demonstrates that static diagnostic benchmarks are insufficient to ensure consistent clinical reliability over time, highlighting the importance of phase-adaptive safeguards in multistage medical AI.