Evaluation of GPT-4 for chest X-ray impression generation: A reader study on performance and perception

📄 arXiv: 2311.06815v1 📥 PDF

作者: Sebastian Ziegelmayer, Alexander W. Marka, Nicolas Lenhart, Nadja Nehls, Stefan Reischl, Felix Harder, Andreas Sauter, Marcus Makowski, Markus Graf, Joshua Gawlitza

分类: cs.CL

发布日期: 2023-11-12

期刊: J Med Internet Res 2023;25:e50865

DOI: 10.2196/50865

💡 一句话要点

评估GPT-4在胸部X光印象生成中的应用以减轻放射科医生负担

🎯 匹配领域: 支柱九：具身大模型 (Embodied Foundation Models)

关键词: 多模态模型 胸部X光 印象生成 GPT-4 放射学 自动化评估 医学影像

📋 核心要点

  1. 现有的放射学印象生成方法在准确性和效率上存在不足，尤其是在处理复杂图像时。
  2. 本研究提出利用GPT-4模型，通过多种输入模态生成胸部X光印象，以提高生成质量和效率。
  3. 实验结果表明，尽管人类撰写的印象评分最高，但AI生成的印象在某些情况下与人类印象相似，且存在偏差问题。

📝 摘要（中文）

本研究探索了多模态基础模型GPT-4在胸部X光印象生成中的能力，旨在减轻放射科医生的工作负担。研究中使用了25个来自NIH数据集的案例，评估了基于不同输入模态（图像、文本、图像与文本结合）生成的印象。结果显示，尽管人类撰写的印象评分最高，但与文本生成的印象差异不显著。AI生成的印象在放射学评分上表现较差，且检测结果存在偏差，表明AI生成的印象可能被误认为是人类撰写的。

🔬 方法详解

问题定义：本研究旨在解决胸部X光印象生成中的准确性和效率问题。现有方法在处理多模态输入时，往往无法充分利用图像和文本信息，导致生成结果的质量不高。

核心思路：通过使用GPT-4模型，结合图像和文本输入，生成更为准确的胸部X光印象。研究设计了多种输入组合，以评估不同模态对生成结果的影响。

技术框架：研究采用了一个包含数据预处理、模型输入、印象生成和评估的整体流程。首先，利用NIH数据集中的图像和文本信息进行数据准备，然后将这些信息输入GPT-4进行印象生成，最后通过放射科医生的评分进行评估。

关键创新：本研究的创新点在于将多模态输入（图像、文本）结合应用于GPT-4模型，探索其在医学影像领域的生成能力。这一方法与传统的单一模态输入方法有本质区别。

关键设计：在模型训练中，采用了特定的损失函数以优化生成印象的质量，并设计了多种输入组合（仅图像、仅文本、图像与文本结合）以评估其对生成结果的影响。

📊 实验亮点

实验结果显示，人类撰写的印象在放射学评分中得分最高，但与文本生成的印象差异不显著。AI生成的印象在放射学评分上表现较差，文本输入的AI生成印象检测率为61%，显示出生成质量的偏差。

🎯 应用场景

该研究的潜在应用领域包括医疗影像分析、放射科自动化报告生成等。通过提高印象生成的准确性和效率，能够显著减轻放射科医生的工作负担，提升医疗服务质量，未来可能推动智能医疗的发展。

📄 摘要（原文）

The remarkable generative capabilities of multimodal foundation models are currently being explored for a variety of applications. Generating radiological impressions is a challenging task that could significantly reduce the workload of radiologists. In our study we explored and analyzed the generative abilities of GPT-4 for Chest X-ray impression generation. To generate and evaluate impressions of chest X-rays based on different input modalities (image, text, text and image), a blinded radiological report was written for 25-cases of the publicly available NIH-dataset. GPT-4 was given image, finding section or both sequentially to generate an input dependent impression. In a blind randomized reading, 4-radiologists rated the impressions and were asked to classify the impression origin (Human, AI), providing justification for their decision. Lastly text model evaluation metrics and their correlation with the radiological score (summation of the 4 dimensions) was assessed. According to the radiological score, the human-written impression was rated highest, although not significantly different to text-based impressions. The automated evaluation metrics showed moderate to substantial correlations to the radiological score for the image impressions, however individual scores were highly divergent among inputs, indicating insufficient representation of radiological quality. Detection of AI-generated impressions varied by input and was 61% for text-based impressions. Impressions classified as AI-generated had significantly worse radiological scores even when written by a radiologist, indicating potential bias. Our study revealed significant discrepancies between a radiological assessment and common automatic evaluation metrics depending on the model input. The detection of AI-generated findings is subject to bias that highly rated impressions are perceived as human-written.