Generalization in Healthcare AI: Evaluation of a Clinical Large Language Model

📄 arXiv: 2402.10965v2 📥 PDF

作者: Salman Rahman, Lavender Yao Jiang, Saadia Gabriel, Yindalon Aphinyanaphongs, Eric Karl Oermann, Rumi Chunara

分类: cs.CL, cs.CY, cs.LG

发布日期: 2024-02-14 (更新: 2024-02-24)

💡 一句话要点

评估临床大语言模型以提升医疗AI的泛化能力

🎯 匹配领域: 支柱九：具身大模型 (Embodied Foundation Models)

关键词: 医疗AI 大语言模型 泛化能力 临床决策 本地微调 患者风险评估 样本量影响

📋 核心要点

  1. 现有的医疗AI模型在不同医院和患者特征下的泛化能力不足，影响了其临床应用效果。
  2. 本文提出通过本地微调等方法来提升大语言模型在医疗领域的泛化能力，尤其是在样本量有限的情况下。
  3. 实验结果表明，本地微调显著提高了模型的AUC，提升幅度可达11.74%，为医疗AI的实际应用提供了新思路。

📝 摘要（中文）

随着大语言模型（LLMs）的进步，为改善患者护理、临床决策和提升医务人员工作流程提供了新机遇。然而，这些模型的潜力在于其在不同临床环境和人群中的有效泛化能力。本文评估了在[HOSPITAL]的临床笔记上训练的ClinicLLM模型，分析其在30天全因再入院预测中的表现，发现样本量少的医院、政府及未指定保险的患者、老年人和高合并症患者的泛化能力较差。通过描述性统计和监督分类，识别出样本量、患者年龄、合并症数量和笔记字数等因素与泛化能力相关。最后，比较了本地微调、基于实例的增强微调和基于聚类的微调，发现本地微调在数据有限的情况下最为有效，AUC提升幅度为0.25%至11.74%。

🔬 方法详解

问题定义：本文旨在解决医疗AI模型在不同临床环境和人群中的泛化能力不足的问题。现有方法往往低估了这一挑战，导致模型在特定医院或患者群体中的表现不佳。

核心思路：通过评估ClinicLLM模型在不同医院和患者特征下的表现，识别影响泛化能力的关键因素，并提出本地微调等方法来改善模型的泛化能力。

技术框架：研究首先分析了样本量、患者特征（如年龄、合并症数量、保险类型）和医院特征对模型性能的影响，然后比较了不同微调策略的效果，包括本地微调、实例增强微调和聚类微调。

关键创新：本研究的创新点在于系统性地分析了影响医疗AI模型泛化能力的多种因素，并提出了本地微调作为有效的解决方案，显著提升了模型在数据稀缺环境下的表现。

关键设计：在实验中，重点关注样本量、患者年龄、合并症数量和笔记字数等参数，采用描述性统计和监督分类方法进行特征识别，确保模型训练的有效性和可靠性。

🖼️ 关键图片

fig_0
fig_1
fig_2

📊 实验亮点

实验结果显示，通过本地微调，模型的AUC提升幅度在0.25%至11.74%之间，尤其在样本量有限的医院环境中表现出显著的效果。这一发现为医疗AI模型的实际应用提供了重要的实证支持。

🎯 应用场景

该研究的潜在应用领域包括医院管理、临床决策支持系统和患者风险评估等。通过提升大语言模型的泛化能力，可以更好地服务于不同患者群体，改善医疗服务质量，最终促进患者健康和医疗效率的提升。

📄 摘要（原文）

Advances in large language models (LLMs) provide new opportunities in healthcare for improved patient care, clinical decision-making, and enhancement of physician and administrator workflows. However, the potential of these models importantly depends on their ability to generalize effectively across clinical environments and populations, a challenge often underestimated in early development. To better understand reasons for these challenges and inform mitigation approaches, we evaluated ClinicLLM, an LLM trained on [HOSPITAL]'s clinical notes, analyzing its performance on 30-day all-cause readmission prediction focusing on variability across hospitals and patient characteristics. We found poorer generalization particularly in hospitals with fewer samples, among patients with government and unspecified insurance, the elderly, and those with high comorbidities. To understand reasons for lack of generalization, we investigated sample sizes for fine-tuning, note content (number of words per note), patient characteristics (comorbidity level, age, insurance type, borough), and health system aspects (hospital, all-cause 30-day readmission, and mortality rates). We used descriptive statistics and supervised classification to identify features. We found that, along with sample size, patient age, number of comorbidities, and the number of words in notes are all important factors related to generalization. Finally, we compared local fine-tuning (hospital specific), instance-based augmented fine-tuning and cluster-based fine-tuning for improving generalization. Among these, local fine-tuning proved most effective, increasing AUC by 0.25% to 11.74% (most helpful in settings with limited data). Overall, this study provides new insights for enhancing the deployment of large language models in the societally important domain of healthcare, and improving their performance for broader populations.