GSCo: Towards Generalizable AI in Medicine via Generalist-Specialist Collaboration

📄 arXiv: 2404.15127v2 📥 PDF

作者: Sunan He, Yuxiang Nie, Hongmei Wang, Shu Yang, Yihui Wang, Zhiyuan Cai, Zhixuan Chen, Yingxue Xu, Luyang Luo, Huiling Xiang, Xi Lin, Mingxiang Wu, Yifan Peng, George Shih, Ziyang Xu, Xian Wu, Qiong Wang, Ronald Cheong Kin Chan, Varut Vardhanabhuti, Winnie Chiu Wing Chu, Yefeng Zheng, Pranav Rajpurkar, Kang Zhang, Hao Chen

分类: cs.CV, cs.CL

发布日期: 2024-04-23 (更新: 2024-11-04)

💡 一句话要点

提出GSCo框架以实现医学领域的通用AI与专家模型协作

🎯 匹配领域: 支柱九：具身大模型 (Embodied Foundation Models)

关键词: 通用基础模型 专家模型 医学图像分析 协同推理 疾病诊断 开源模型 轻量级模型

📋 核心要点

  1. 现有医学AI模型往往是单一的通用模型或专家模型，难以兼顾广泛性和精准性。
  2. 本文提出GSCo框架，通过构建通用基础模型与专家模型的协作，实现医学图像分析的精确性与广泛性。
  3. 实验结果显示，MedDr在多个下游任务中表现优异，GSCo在跨域疾病诊断上超越了现有模型，展示了显著的性能提升。

📝 摘要（中文）

通用基础模型（GFM）因其在多任务和多模态中的卓越能力而受到广泛关注。在医学领域，尽管GFM在广泛知识和指令跟随能力上表现出色，但专家模型在精准度上更具优势。本文首次探索GFM与专家模型之间的协同作用，提出了通用-专家协作框架（GSCo），包括GFM和专家模型的构建及下游任务的协同推理。我们开发了MedDr，这是针对医学领域的最大开源GFM，展示了出色的指令跟随和上下文学习能力。同时，为下游任务设计了一系列轻量级专家模型。在协同推理阶段，提出了专家混合诊断和检索增强诊断两种机制，以结合通用模型的上下文学习能力和专家的领域知识。实验结果表明，MedDr在下游数据集上持续超越现有最先进的GFM，GSCo在所有跨域疾病诊断数据集上也优于GFM和专家模型，标志着医学领域通用AI应用的重大转变。

🔬 方法详解

问题定义：本文旨在解决医学领域中通用基础模型与专家模型之间的协同不足问题。现有方法往往无法同时兼顾广泛性和精准性，导致在特定任务中的表现不佳。

核心思路：GSCo框架通过结合通用基础模型的广泛知识和专家模型的领域专长，形成协同推理机制，以提高医学图像分析的准确性和效率。

技术框架：GSCo框架分为两个主要阶段：第一阶段是构建MedDr（针对医学的开源GFM）和轻量级专家模型；第二阶段是进行协同推理，利用专家混合诊断和检索增强诊断机制。

关键创新：GSCo框架的创新在于首次将GFM与专家模型结合，通过协同推理机制实现了更高的诊断精度，突破了传统模型的局限。

关键设计：在模型构建中，MedDr采用了先进的指令跟随和上下文学习技术，专家模型则设计为轻量级以降低计算成本。协同推理阶段引入的机制能够有效整合两者的优势，提升整体性能。

🖼️ 关键图片

fig_0
fig_1
fig_2

📊 实验亮点

实验结果表明，MedDr在下游任务中表现优于现有最先进的GFM，且GSCo在所有跨域疾病诊断数据集上均超越了GFM和专家模型，显示出显著的性能提升，具体提升幅度未知。

🎯 应用场景

该研究在医学图像分析、疾病诊断等领域具有广泛的应用潜力。通过GSCo框架，医疗机构能够更高效地利用AI技术进行精准诊断，提升医疗服务质量。此外，未来可扩展至其他领域的通用AI应用，推动智能医疗的发展。

📄 摘要（原文）

Generalist foundation models (GFMs) are renowned for their exceptional capability and flexibility in effectively generalizing across diverse tasks and modalities. In the field of medicine, while GFMs exhibit superior generalizability based on their extensive intrinsic knowledge as well as proficiency in instruction following and in-context learning, specialist models excel in precision due to their domain knowledge. In this work, for the first time, we explore the synergy between the GFM and specialist models, to enable precise medical image analysis on a broader scope. Specifically, we propose a cooperative framework, Generalist-Specialist Collaboration (GSCo), which consists of two stages, namely the construction of GFM and specialists, and collaborative inference on downstream tasks. In the construction stage, we develop MedDr, the largest open-source GFM tailored for medicine, showcasing exceptional instruction-following and in-context learning capabilities. Meanwhile, a series of lightweight specialists are crafted for downstream tasks with low computational cost. In the collaborative inference stage, we introduce two cooperative mechanisms, Mixture-of-Expert Diagnosis and Retrieval-Augmented Diagnosis, to harvest the generalist's in-context learning abilities alongside the specialists' domain expertise. For a comprehensive evaluation, we curate a large-scale benchmark featuring 28 datasets and about 250,000 images. Extensive results demonstrate that MedDr consistently outperforms state-of-the-art GFMs on downstream datasets. Furthermore, GSCo exceeds both GFMs and specialists across all out-of-domain disease diagnosis datasets. These findings indicate a significant paradigm shift in the application of GFMs, transitioning from separate models for specific tasks to a collaborative approach between GFMs and specialists, thereby advancing the frontiers of generalizable AI in medicine.