Development of a Reliable and Accessible Caregiving Language Model (CaLM)

📄 arXiv: 2403.06857v1 📥 PDF

作者: Bambang Parmanto, Bayu Aryoyudanta, Wilbert Soekinto, I Made Agus Setiawan, Yuhan Wang, Haomin Hu, Andi Saptono, Yong K. Choi

分类: cs.CL

发布日期: 2024-03-11

💡 一句话要点

提出可靠且易于获取的护理语言模型以支持家庭护理者

🎯 匹配领域: 支柱九：具身大模型 (Embodied Foundation Models)

关键词: 护理语言模型 家庭护理 检索增强生成 小型语言模型 知识库构建 微调技术 性能评估

📋 核心要点

  1. 家庭护理者缺乏专业培训，导致护理质量参差不齐，亟需提升其能力。
  2. 研究提出通过小型语言模型结合护理知识库，开发可靠的护理语言模型（CaLM）。
  3. 实验结果显示，经过RAG微调的小型模型在多个指标上超越了大型模型GPT-3.5，表现出色。

📝 摘要（中文）

家庭护理者通常缺乏专业培训，因此提升其护理能力至关重要。本文开发了一种可靠的护理语言模型（CaLM），利用大型语言模型作为基础技术，支持家庭护理者的教育和护理辅助。研究采用检索增强生成（RAG）框架，结合小型语言模型进行微调，以提高模型回答的质量，并建立了一个护理知识库。通过与大型语言模型GPT-3.5的对比，结果表明，经过RAG微调的小型语言模型在多个评估指标上表现优于大型模型，显示出小型模型在特定领域的可靠性和可获取性。

🔬 方法详解

问题定义：本文旨在解决家庭护理者在缺乏专业培训情况下的护理能力不足问题。现有方法多依赖大型语言模型，计算资源需求高，难以普及。

核心思路：研究通过结合小型语言模型与护理知识库，利用检索增强生成（RAG）框架，提升模型的回答质量，使其更适合家庭护理者使用。

技术框架：整体架构包括数据收集、知识库构建、模型训练和评估四个主要模块。首先从互联网收集相关文档构建护理知识库，然后对小型语言模型进行微调，最后通过标准评估指标对模型性能进行测试。

关键创新：本研究的创新点在于通过RAG框架显著提升小型语言模型的性能，使其在特定护理领域的应用中表现优于大型模型，打破了对大型模型的依赖。

关键设计：在模型训练中，选择了LLaMA-2和Falcon作为小型模型候选，使用7B参数进行微调，设计了特定的损失函数以优化模型的回答准确性和参考文献的返回能力。

📊 实验亮点

实验结果显示，经过RAG微调的LLaMA-2小型模型在所有评估指标上均优于GPT-3.5，尤其在返回参考文献的准确性上表现突出，展示了小型模型在特定领域的强大能力。

🎯 应用场景

该研究的潜在应用领域包括家庭护理、老年人护理和慢性病管理等。通过提供易于获取的护理语言模型，家庭护理者能够获得更好的支持，从而提升护理质量，减轻专业护理人员的负担，具有重要的社会价值和实际意义。

📄 摘要（原文）

Unlike professional caregivers, family caregivers often assume this role without formal preparation or training. Because of this, there is an urgent need to enhance the capacity of family caregivers to provide quality care. Large language models can potentially be used as a foundation technology for supporting caregivers as educational tools or as adjunct to care. This study aimed to develop a reliable Caregiving Language Model (CaLM) by using FMs and a caregiving knowledge base, develop an accessible CaLM using a small FM that requires fewer computing resources, and evaluate the performance of the model compared to a large FM. We developed CaLM using the Retrieval Augmented Generation (RAG) framework combined with FM fine-tuning for improving the quality of FM answers by grounding the model on a caregiving knowledge base. We used two small FMs as candidates for the FM of CaLM (LLaMA-2 and Falcon with 7B parameters) and larger FM GPT-3.5 as a benchmark. We developed the caregiving knowledge base by gathering various types of documents from the Internet. In this study, we focused on caregivers of individuals with Alzheimer's Disease Related Dementias. We evaluated the models' performance using the benchmark metrics commonly used in evaluating language models and their reliability to provide accurate references with the answers. The RAG framework improved the performance of all FMs used in this study across all measures. As expected, the large FM performed better than small FMs across all metrics. The most interesting result is that small fine-tuned FMs with RAG performed significantly better than GPT 3.5 across all metrics. The fine-tuned LLaMA-2 small FM performed better than GPT 3.5 (even with RAG) in returning references with the answers. The study shows that reliable and accessible CaLM can be developed by using small FMs with a knowledge base specific to the caregiving domain.