RJUA-MedDQA: A Multimodal Benchmark for Medical Document Question Answering and Clinical Reasoning

📄 arXiv: 2402.14840v1 📥 PDF

作者: Congyun Jin, Ming Zhang, Xiaowei Ma, Li Yujiao, Yingbo Wang, Yabo Jia, Yuliang Du, Tao Sun, Haowen Wang, Cong Fan, Jinjie Gu, Chenfei Chi, Xiangguo Lv, Fangzhou Li, Wei Xue, Yiran Huang

分类: cs.CL, cs.AI, stat.AP

发布日期: 2024-02-19

备注: 15 pages, 13 figures

💡 一句话要点

提出RJUA-MedDQA以解决医疗文档问答与临床推理问题

🎯 匹配领域: 支柱九：具身大模型 (Embodied Foundation Models)

关键词: 医疗文档问答 临床推理 多模态模型 图像内容解读 数值推理 高效注释 智能医疗诊断

📋 核心要点

  1. 现有医疗文档问答基准未能充分反映真实医疗报告的复杂性和推理能力，存在显著不足。
  2. 提出RJUA-MedDQA基准，设计高效结构恢复注释（ESRA）方法，旨在提高医疗报告图像的注释效率和准确性。
  3. 通过对5个大型多模态模型的评估，发现LMM在低质量图像上表现更稳健，整体性能仍有限，推理能力面临挑战。

📝 摘要（中文）

近年来，大型语言模型（LLMs）和大型多模态模型（LMMs）在智能医疗诊断等多个医疗应用中展现出潜力。然而，现有基准未能反映真实医疗报告的复杂性及深入推理能力。为此，本文提出了RJUA-MedDQA，一个全面的医疗专业基准，旨在解决图像内容解读、数值推理和临床推理等多项挑战。我们设计了高效结构恢复注释（ESRA）方法，显著提高了注释效率，提升了26.8%的准确率。通过对5个LMM的少量样本评估，我们发现现有LMM在低质量和多样化结构图像上表现更为稳健，但在上下文和图像内容推理方面仍面临重大挑战。希望该基准能推动多模态医疗文档理解的进展。

🔬 方法详解

问题定义：本文旨在解决现有医疗文档问答基准在复杂性和推理能力上的不足，特别是如何有效解读医疗报告中的图像内容和进行临床推理。

核心思路：提出RJUA-MedDQA基准，并设计高效结构恢复注释（ESRA）方法，以提高医疗报告图像的注释效率和准确性，进而推动多模态医疗文档理解。

技术框架：整体架构包括数据生成管道和ESRA方法，主要模块包括图像内容解读、数值推理和临床推理能力的评估。

关键创新：ESRA方法是本文的核心创新，通过恢复医疗报告图像中的文本和表格内容，显著提高了注释效率和准确性，较现有方法有本质区别。

关键设计：在ESRA方法中，采用了特定的参数设置和损失函数，以优化图像内容的恢复效果，确保注释的高效性和准确性。具体的网络结构和技术细节在论文中进行了详细描述。

🖼️ 关键图片

fig_0
fig_1

📊 实验亮点

实验结果显示，ESRA方法使注释效率翻倍，准确率提升26.8%。在对5个LMM的评估中，发现LMM在处理低质量和多样化结构图像时表现更为稳健，但整体性能仍有限，推理能力面临显著挑战。

🎯 应用场景

RJUA-MedDQA基准的提出为医疗文档问答和临床推理提供了新的研究方向，具有广泛的应用潜力。该基准可用于提升医疗AI系统在实际医疗场景中的表现，促进智能医疗诊断的发展，未来可能对医疗行业产生深远影响。

📄 摘要（原文）

Recent advancements in Large Language Models (LLMs) and Large Multi-modal Models (LMMs) have shown potential in various medical applications, such as Intelligent Medical Diagnosis. Although impressive results have been achieved, we find that existing benchmarks do not reflect the complexity of real medical reports and specialized in-depth reasoning capabilities. In this work, we introduced RJUA-MedDQA, a comprehensive benchmark in the field of medical specialization, which poses several challenges: comprehensively interpreting imgage content across diverse challenging layouts, possessing numerical reasoning ability to identify abnormal indicators and demonstrating clinical reasoning ability to provide statements of disease diagnosis, status and advice based on medical contexts. We carefully design the data generation pipeline and proposed the Efficient Structural Restoration Annotation (ESRA) Method, aimed at restoring textual and tabular content in medical report images. This method substantially enhances annotation efficiency, doubling the productivity of each annotator, and yields a 26.8% improvement in accuracy. We conduct extensive evaluations, including few-shot assessments of 5 LMMs which are capable of solving Chinese medical QA tasks. To further investigate the limitations and potential of current LMMs, we conduct comparative experiments on a set of strong LLMs by using image-text generated by ESRA method. We report the performance of baselines and offer several observations: (1) The overall performance of existing LMMs is still limited; however LMMs more robust to low-quality and diverse-structured images compared to LLMs. (3) Reasoning across context and image content present significant challenges. We hope this benchmark helps the community make progress on these challenging tasks in multi-modal medical document understanding and facilitate its application in healthcare.