A Multi-Center Study on the Adaptability of a Shared Foundation Model for Electronic Health Records

📄 arXiv: 2311.11483v2 📥 PDF

作者: Lin Lawrence Guo, Jason Fries, Ethan Steinberg, Scott Lanyon Fleming, Keith Morse, Catherine Aftandilian, Jose Posada, Nigam Shah, Lillian Sung

分类: cs.LG, cs.AI

发布日期: 2023-11-20 (更新: 2024-04-23)

备注: 46 pages, 5 figures, 3 tables, 14 appendices

💡 一句话要点

提出共享基础模型以提升电子健康记录的适应性

🎯 匹配领域: 支柱九：具身大模型 (Embodied Foundation Models)

关键词: 基础模型 电子健康记录 医疗AI 多中心研究 模型适应性 预训练 预测性能 标签效率

📋 核心要点

  1. 现有的EHR基础模型在不同医院间共享时，适应性和性能仍存在不确定性，限制了其广泛应用。
  2. 本文提出通过在本地数据上继续预训练共享EHR基础模型，来提高其在特定任务上的适应性和性能。
  3. 实验结果显示，适应共享模型的性能与本地训练模型相当，并在少量训练标签的情况下提升了13%的预测准确率。

📝 摘要（中文）

基础模型在医疗领域的应用前景广阔，能够提供模块化组件，便于适应下游任务，从而使AI开发更具可扩展性和成本效益。本文通过多中心研究，探讨了一个新发布的结构化电子健康记录（EHR）基础模型在不同医院的适应性。研究表明，该模型在少量任务特定标签的情况下，能够与本地训练的模型性能相匹配，并在继续预训练时显著提高标签效率，显示出共享EHR基础模型的潜力，能够以更低的成本提升预测性能。

🔬 方法详解

问题定义：本文旨在解决共享基础模型在不同医院间适应性不足的问题，现有方法在不同数据分布下的性能表现不佳，限制了模型的推广应用。

核心思路：通过在本地数据上继续预训练共享的EHR基础模型，增强其对特定任务的适应性，从而提高模型的性能和效率。

技术框架：研究采用了多中心实验设计，使用来自斯坦福医学中心的2.57M患者的纵向医疗记录训练的基础模型（$FM_{SM}$），并在其他医院的数据上进行适应性测试。主要模块包括模型预训练、任务适应性评估和性能比较。

关键创新：该研究的创新在于展示了共享EHR基础模型在不同医院间的适应性，通过继续预训练显著提高了模型的标签效率和样本利用率，突破了传统模型训练的局限。

关键设计：在实验中，$FM_{SM}$模型在本地数据上继续预训练，使用了少于1%的训练样本便能达到完全训练的GBM模型的性能，且在样本效率上比从头训练本地基础模型提高了60%至90%。

📊 实验亮点

实验结果显示，适应共享的$FM_{SM}$模型在两个数据集上均达到了与本地训练的GBM模型相当的性能，且在少量任务特定标签的情况下提升了13%。继续预训练的样本效率比从头训练本地模型提高了60%至90%。

🎯 应用场景

该研究的成果在医疗AI领域具有广泛的应用潜力，能够帮助不同医院快速部署高效的预测模型，降低开发成本，提高医疗服务的质量和效率。未来，随着更多医院采用共享基础模型，可能会推动医疗数据的整合与共享，促进个性化医疗的发展。

📄 摘要（原文）

Foundation models hold promise for transforming AI in healthcare by providing modular components that are easily adaptable to downstream healthcare tasks, making AI development more scalable and cost-effective. Structured EHR foundation models, trained on coded medical records from millions of patients, demonstrated benefits including increased performance with fewer training labels, and improved robustness to distribution shifts. However, questions remain on the feasibility of sharing these models across different hospitals and their performance for local task adaptation. This multi-center study examined the adaptability of a recently released structured EHR foundation model ($FM_{SM}$), trained on longitudinal medical record data from 2.57M Stanford Medicine patients. Experiments were conducted using EHR data at The Hospital for Sick Children and MIMIC-IV. We assessed both adaptability via continued pretraining on local data, and task adaptability compared to baselines of training models from scratch at each site, including a local foundation model. We evaluated the performance of these models on 8 clinical prediction tasks. In both datasets, adapting the off-the-shelf $FM_{SM}$ matched the performance of GBM models locally trained on all data while providing a 13% improvement in settings with few task-specific training labels. With continued pretraining on local data, label efficiency substantially improved, such that $FM_{SM}$ required fewer than 1% of training examples to match the fully trained GBM's performance. Continued pretraining was also 60 to 90% more sample-efficient than training local foundation models from scratch. Our findings show that adapting shared EHR foundation models across hospitals provides improved prediction performance at less cost, underscoring the utility of base foundation models as modular components to streamline the development of healthcare AI.