Real-Time Multimodal Cognitive Assistant for Emergency Medical Services

📄 arXiv: 2403.06734v1 📥 PDF

作者: Keshara Weerasinghe, Saahith Janapati, Xueren Ge, Sion Kim, Sneha Iyer, John A. Stankovic, Homa Alemzadeh

分类: cs.AI, cs.CL, cs.CV

发布日期: 2024-03-11

备注: This work has been submitted to the IEEE for possible publication

💡 一句话要点

提出CognitiveEMS以解决紧急医疗服务中的认知负荷问题

🎯 匹配领域: 支柱九：具身大模型 (Embodied Foundation Models)

关键词: 紧急医疗服务 认知助手 多模态数据 语音识别 协议预测 动作识别 边缘计算 增强现实

📋 核心要点

  1. 现有的紧急医疗服务响应者在高压环境下容易出现认知过载，影响决策效率。
  2. CognitiveEMS系统通过实时数据获取与分析，结合AR技术，提供智能化的决策支持。
  3. 实验结果表明，语音识别和协议预测的性能显著优于现有技术，动作识别准确率达到0.727。

📝 摘要（中文）

紧急医疗服务（EMS）响应者常在时间紧迫的条件下工作，面临认知过载和固有风险，需具备关键的批判性思维和快速决策能力。本文提出CognitiveEMS，一个端到端的可穿戴认知助手系统，能够作为协作虚拟伙伴，实时获取和分析来自紧急现场的多模态数据，并通过增强现实（AR）智能眼镜与EMS响应者互动。CognitiveEMS实时处理数据流，利用边缘计算提供EMS协议选择和干预识别的支持。我们通过引入三个新颖组件解决实时认知辅助中的关键技术挑战：针对真实医疗紧急对话微调的语音识别模型、结合最新小型语言模型与EMS领域知识的EMS协议预测模型，以及利用多模态音频和视频数据推断响应者干预动作的EMS动作识别模块。我们的结果显示，语音识别在对话数据上表现优于现有技术，协议预测组件的准确率显著提升，动作识别准确率达到0.727。

🔬 方法详解

问题定义：本研究旨在解决紧急医疗服务中响应者面临的认知负荷和决策延迟问题。现有方法在实时数据处理和决策支持方面存在不足，无法有效应对复杂的紧急情况。

核心思路：CognitiveEMS系统通过集成多模态数据处理、边缘计算和AR技术，提供实时的认知辅助，帮助EMS响应者快速做出决策。设计上强调实时性和准确性，以应对紧急医疗场景的需求。

技术框架：系统主要包括三个模块：语音识别模型、EMS协议预测模型和EMS动作识别模块。语音识别模块处理医疗对话，协议预测模块结合EMS知识进行决策支持，动作识别模块则分析响应者的实际干预行为。

关键创新：本研究的创新点在于引入了针对真实医疗对话的语音识别模型和结合图注意力机制的协议预测模型，显著提升了系统在复杂场景下的表现。

关键设计：语音识别模型通过模拟EMS音频录音进行微调，并使用大型语言模型生成的合成数据增强训练；协议预测模型结合最新的小型语言模型和EMS领域知识，采用图结构进行信息处理。

🖼️ 关键图片

fig_0
fig_1
fig_2

📊 实验亮点

实验结果显示，语音识别模型在对话数据上的字错误率（WER）为0.290，显著优于现有技术的0.618；协议预测的前3名准确率达到0.800，而现有技术仅为0.200；动作识别模块的准确率为0.727，整体延迟保持在3.78秒（边缘）和0.31秒（服务器）。

🎯 应用场景

CognitiveEMS系统具有广泛的应用潜力，尤其在紧急医疗服务、灾难响应和其他需要快速决策的领域。通过提供实时的认知支持，该系统能够显著提高响应者的工作效率和决策质量，未来可能在医疗、公共安全等多个领域产生深远影响。

📄 摘要（原文）

Emergency Medical Services (EMS) responders often operate under time-sensitive conditions, facing cognitive overload and inherent risks, requiring essential skills in critical thinking and rapid decision-making. This paper presents CognitiveEMS, an end-to-end wearable cognitive assistant system that can act as a collaborative virtual partner engaging in the real-time acquisition and analysis of multimodal data from an emergency scene and interacting with EMS responders through Augmented Reality (AR) smart glasses. CognitiveEMS processes the continuous streams of data in real-time and leverages edge computing to provide assistance in EMS protocol selection and intervention recognition. We address key technical challenges in real-time cognitive assistance by introducing three novel components: (i) a Speech Recognition model that is fine-tuned for real-world medical emergency conversations using simulated EMS audio recordings, augmented with synthetic data generated by large language models (LLMs); (ii) an EMS Protocol Prediction model that combines state-of-the-art (SOTA) tiny language models with EMS domain knowledge using graph-based attention mechanisms; (iii) an EMS Action Recognition module which leverages multimodal audio and video data and protocol predictions to infer the intervention/treatment actions taken by the responders at the incident scene. Our results show that for speech recognition we achieve superior performance compared to SOTA (WER of 0.290 vs. 0.618) on conversational data. Our protocol prediction component also significantly outperforms SOTA (top-3 accuracy of 0.800 vs. 0.200) and the action recognition achieves an accuracy of 0.727, while maintaining an end-to-end latency of 3.78s for protocol prediction on the edge and 0.31s on the server.