OpenMedLM: Prompt engineering can out-perform fine-tuning in medical question-answering with open-source large language models

📄 arXiv: 2402.19371v1 📥 PDF

作者: Jenish Maharjan, Anurag Garikipati, Navan Preet Singh, Leo Cyrus, Mayank Sharma, Madalina Ciobanu, Gina Barnes, Rahul Thapa, Qingqing Mao, Ritankar Das

分类: cs.CL, cs.AI, cs.IR

发布日期: 2024-02-29

💡 一句话要点

提出OpenMedLM以提升医疗问答的开放模型性能

🎯 匹配领域: 支柱九：具身大模型 (Embodied Foundation Models)

关键词: 大型语言模型 医疗问答 提示工程 开放源代码 性能提升 基准测试 计算成本

📋 核心要点

  1. 现有医疗LLMs通常依赖于昂贵的微调过程，限制了其可访问性和透明性。
  2. OpenMedLM通过提示工程策略，提供了一种高效的替代方案，避免了传统微调的高成本。
  3. 实验结果显示，OpenMedLM在多个医疗基准测试中取得了显著的性能提升，尤其在MedQA和MMLU医疗子集上表现突出。

📝 摘要（中文）

大型语言模型（LLMs）在执行多种专业任务方面日益强大，能够扩展医疗知识的公平获取。大多数医疗LLMs依赖于大量的微调，需耗费昂贵的计算资源。OpenMedLM是一个提示工程平台，针对开放源代码LLMs在医疗基准测试中提供了最先进的性能。通过多种提示策略，OpenMedLM在多个医疗基准测试中超越了以往依赖于计算密集型微调的最佳开放模型，展现了医疗特定的突现特性，并强调了提示工程在提升可访问LLMs性能中的重要性。

🔬 方法详解

问题定义：本论文旨在解决医疗问答领域中开放源代码大型语言模型（LLMs）性能不足的问题。现有方法依赖于昂贵的微调过程，限制了模型的可访问性和透明性。

核心思路：论文提出的核心思路是通过提示工程（prompt engineering）来提升开放源代码LLMs的性能，避免了传统微调所需的高计算成本。通过多种提示策略，OpenMedLM能够有效地利用现有的开放模型。

技术框架：OpenMedLM的整体架构包括多个模块，首先是基础模型的选择（7B-70B的开放源代码LLMs），然后应用不同的提示策略，如零-shot、few-shot、链式思维（chain-of-thought）和集成/自一致性投票。

关键创新：最重要的技术创新点在于通过提示工程实现了开放源代码LLMs的最先进性能，特别是在医疗基准测试中超越了以往依赖于微调的模型。与现有方法相比，OpenMedLM展示了更高的准确性和更低的计算成本。

关键设计：在实验中，模型的提示策略包括随机选择和kNN选择，此外还采用了集成投票机制。模型在MedQA基准测试中达到了72.6%的准确率，超越了之前的最佳记录2.4%。

📊 实验亮点

OpenMedLM在多个医疗基准测试中取得了显著的实验结果，特别是在MedQA基准测试中达到了72.6%的准确率，超越了之前的最佳开放模型2.4%。在MMLU医疗子集上，OpenMedLM首次实现了超过80%的准确率，标志着开放源代码LLMs在医疗问答领域的重大突破。

🎯 应用场景

OpenMedLM的研究成果在医疗领域具有广泛的应用潜力，能够为医生和患者提供更为高效和准确的医疗问答服务。通过降低模型的使用成本和提高可访问性，该平台可以促进医疗知识的传播，提升医疗服务的公平性和透明度。未来，OpenMedLM有望在临床决策支持、医疗教育和公共健康等多个领域发挥重要作用。

📄 摘要（原文）

LLMs have become increasingly capable at accomplishing a range of specialized-tasks and can be utilized to expand equitable access to medical knowledge. Most medical LLMs have involved extensive fine-tuning, leveraging specialized medical data and significant, thus costly, amounts of computational power. Many of the top performing LLMs are proprietary and their access is limited to very few research groups. However, open-source (OS) models represent a key area of growth for medical LLMs due to significant improvements in performance and an inherent ability to provide the transparency and compliance required in healthcare. We present OpenMedLM, a prompting platform which delivers state-of-the-art (SOTA) performance for OS LLMs on medical benchmarks. We evaluated a range of OS foundation LLMs (7B-70B) on four medical benchmarks (MedQA, MedMCQA, PubMedQA, MMLU medical-subset). We employed a series of prompting strategies, including zero-shot, few-shot, chain-of-thought (random selection and kNN selection), and ensemble/self-consistency voting. We found that OpenMedLM delivers OS SOTA results on three common medical LLM benchmarks, surpassing the previous best performing OS models that leveraged computationally costly extensive fine-tuning. The model delivers a 72.6% accuracy on the MedQA benchmark, outperforming the previous SOTA by 2.4%, and achieves 81.7% accuracy on the MMLU medical-subset, establishing itself as the first OS LLM to surpass 80% accuracy on this benchmark. Our results highlight medical-specific emergent properties in OS LLMs which have not yet been documented to date elsewhere, and showcase the benefits of further leveraging prompt engineering to improve the performance of accessible LLMs for medical applications.