Quality of Answers of Generative Large Language Models vs Peer Patients for Interpreting Lab Test Results for Lay Patients: Evaluation Study

📄 arXiv: 2402.01693v1 📥 PDF

作者: Zhe He, Balu Bhasuran, Qiao Jin, Shubo Tian, Karim Hanna, Cindy Shavor, Lisbeth Garcia Arguello, Patrick Murray, Zhiyong Lu

分类: cs.CL, cs.AI

发布日期: 2024-01-23

DOI: 10.2196/56655

💡 一句话要点

评估生成大型语言模型在实验室测试结果解读中的应用效果

🎯 匹配领域: 支柱九：具身大模型 (Embodied Foundation Models)

关键词: 大型语言模型 医疗问答 实验室测试 患者教育 自然语言处理 模型评估 GPT-4 健康信息

📋 核心要点

  1. 现有的实验室测试结果解读方法常常让患者感到困惑，缺乏准确和个性化的解释。
  2. 本文通过使用大型语言模型生成实验室测试相关问题的回答，探索其在患者教育中的可行性和有效性。
  3. 实验结果表明，GPT-4在相关性、准确性、帮助性和安全性方面均优于其他LLMs和人类回答，显示出其在医疗领域的潜力。

📝 摘要（中文）

实验室结果常常令人困惑且难以理解。大型语言模型（LLMs）如ChatGPT为患者提供了回答问题的新途径。本文旨在评估LLMs生成与实验室测试相关问题的相关性、准确性、帮助性和安全性，并识别可通过增强方法缓解的潜在问题。研究中从Yahoo! Answers收集了实验室测试结果相关的问答数据，选取53对问答进行分析。使用LangChain框架和ChatGPT生成四种LLMs（GPT-4、Meta LLaMA 2、MedAlpaca和ORCA_mini）的回答。通过标准的问答相似性评估指标和LLM评估器对回答质量进行评估，结果显示GPT-4在所有四个方面的表现优于其他模型和人类回答。尽管如此，LLMs的回答仍存在缺乏医学背景解释和不准确的情况。研究提出了多种改进LLM回答质量的方法。

🔬 方法详解

问题定义：本文旨在解决患者在理解实验室测试结果时面临的困惑，现有方法往往缺乏准确性和个性化的解释。

核心思路：通过生成大型语言模型的回答，评估其在解读实验室测试结果中的有效性，探索如何提高回答的质量。

技术框架：研究使用LangChain框架，结合ChatGPT生成四种LLMs的回答，并通过标准评估指标和专家评审进行质量评估。

关键创新：本文的创新在于系统性地比较不同LLMs在医疗问答中的表现，特别是GPT-4在多个评估维度上的优势。

关键设计：研究中使用了ROUGE、BLEU、METEOR和BERTScore等标准评估指标，并结合LLM评估器和医学专家的手动评估，确保结果的全面性和准确性。

📊 实验亮点

实验结果显示，GPT-4在相关性、准确性、帮助性和安全性方面的得分均优于其他三种LLMs和人类回答，尤其在所有四个方面的表现均显著提升，显示出其在医疗问答中的优越性。

🎯 应用场景

该研究的潜在应用领域包括患者教育、医疗咨询和健康信息传播。通过提高LLMs在解读实验室测试结果中的表现，可以帮助患者更好地理解自身健康状况，促进医患沟通，提升医疗服务质量。未来，随着技术的进步，LLMs可能在个性化医疗和智能健康助手中发挥更大作用。

📄 摘要（原文）

Lab results are often confusing and hard to understand. Large language models (LLMs) such as ChatGPT have opened a promising avenue for patients to get their questions answered. We aim to assess the feasibility of using LLMs to generate relevant, accurate, helpful, and unharmful responses to lab test-related questions asked by patients and to identify potential issues that can be mitigated with augmentation approaches. We first collected lab test results related question and answer data from Yahoo! Answers and selected 53 QA pairs for this study. Using the LangChain framework and ChatGPT web portal, we generated responses to the 53 questions from four LLMs including GPT-4, Meta LLaMA 2, MedAlpaca, and ORCA_mini. We first assessed the similarity of their answers using standard QA similarity-based evaluation metrics including ROUGE, BLEU, METEOR, BERTScore. We also utilized an LLM-based evaluator to judge whether a target model has higher quality in terms of relevance, correctness, helpfulness, and safety than the baseline model. Finally, we performed a manual evaluation with medical experts for all the responses to seven selected questions on the same four aspects. The results of Win Rate and medical expert evaluation both showed that GPT-4's responses achieved better scores than all the other LLM responses and human responses on all four aspects (relevance, correctness, helpfulness, and safety). However, LLM responses occasionally also suffer from a lack of interpretation in one's medical context, incorrect statements, and lack of references. We find that compared to other three LLMs and human answer from the Q&A website, GPT-4's responses are more accurate, helpful, relevant, and safer. However, there are cases which GPT-4 responses are inaccurate and not individualized. We identified a number of ways to improve the quality of LLM responses.