Spiking-PhysFormer: Camera-Based Remote Photoplethysmography with Parallel Spike-driven Transformer

📄 arXiv: 2402.04798v4 📥 PDF

作者: Mingxuan Liu, Jiankai Tang, Yongli Chen, Haoxiang Li, Jiahao Qi, Siwei Li, Kegang Wang, Jie Gan, Yuntao Wang, Hong Chen

分类: cs.CV

发布日期: 2024-02-07 (更新: 2025-01-03)

备注: Mingxuan Liu and Jiankai Tang are co-first authors of the article. Accepted by Neural Networks

💡 一句话要点

提出Spiking-PhysFormer以解决移动设备上rPPG的能耗问题

🎯 匹配领域: 支柱八：物理动画 (Physics-based Animation)

关键词: 远程光电容积描记法 脉冲神经网络 能效优化 生理信号监测 深度学习

📋 核心要点

  1. 现有的ANN方法在移动设备上部署时面临高计算资源需求，限制了其应用。
  2. 提出的Spiking-PhysFormer模型结合了ANN和SNN的优点，旨在降低功耗并提高效率。
  3. 实验结果显示，Spiking-PhysFormer在功耗方面比PhysFormer降低了12.4%，并且变换器块的功耗降低了12.2倍。

📝 摘要（中文）

人工神经网络（ANNs）能够提高基于相机的远程光电容积描记法（rPPG）在测量心脏活动和生理信号方面的准确性。然而，现有的ANN方法通常需要大量计算资源，这对移动设备的有效部署构成挑战。本文首次将脉冲神经网络（SNNs）引入rPPG领域，提出了一种混合神经网络模型Spiking-PhysFormer，旨在降低功耗。该模型由基于ANN的补丁嵌入块、基于SNN的变换器块和基于ANN的预测头组成。实验结果表明，该模型在四个数据集上实现了12.4%的功耗降低，同时保持了与PhysFormer和其他ANN模型相当的性能。

🔬 方法详解

问题定义：本文旨在解决基于相机的远程光电容积描记法（rPPG）在移动设备上应用时的高能耗问题。现有的ANN方法虽然准确，但计算资源需求高，限制了其在移动设备上的有效部署。

核心思路：论文提出的Spiking-PhysFormer模型通过结合脉冲神经网络（SNN）和人工神经网络（ANN），实现了能效的显著提升。通过设计并行脉冲变换器块，简化了传统变换器的结构，同时保留了其聚合时空特征的能力。

技术框架：Spiking-PhysFormer的整体架构包括三个主要模块：基于ANN的补丁嵌入块、基于SNN的变换器块和基于ANN的预测头。补丁嵌入块负责将输入图像转换为特征表示，变换器块负责特征的处理与聚合，预测头则输出最终的生理信号。

关键创新：最重要的技术创新在于首次将SNN引入rPPG领域，并设计了并行脉冲变换器块和简化的脉冲自注意力机制，显著降低了功耗而不影响性能。

关键设计：模型中采用了简化的脉冲自注意力机制，省略了值参数的使用，确保了模型在保持性能的同时实现了功耗的降低。

🖼️ 关键图片

fig_0
fig_1
fig_2

📊 实验亮点

实验结果表明，Spiking-PhysFormer在四个数据集上实现了12.4%的功耗降低，相较于PhysFormer，变换器块的功耗降低了12.2倍，同时保持了与其他ANN模型相当的性能，展示了其在能效和准确性上的优势。

🎯 应用场景

该研究的潜在应用领域包括健康监测、远程医疗和智能穿戴设备等。通过降低功耗，Spiking-PhysFormer能够在移动设备上实现更高效的生理信号监测，推动可穿戴技术的发展，提升用户体验和健康管理的便利性。

📄 摘要（原文）

Artificial neural networks (ANNs) can help camera-based remote photoplethysmography (rPPG) in measuring cardiac activity and physiological signals from facial videos, such as pulse wave, heart rate and respiration rate with better accuracy. However, most existing ANN-based methods require substantial computing resources, which poses challenges for effective deployment on mobile devices. Spiking neural networks (SNNs), on the other hand, hold immense potential for energy-efficient deep learning owing to their binary and event-driven architecture. To the best of our knowledge, we are the first to introduce SNNs into the realm of rPPG, proposing a hybrid neural network (HNN) model, the Spiking-PhysFormer, aimed at reducing power consumption. Specifically, the proposed Spiking-PhyFormer consists of an ANN-based patch embedding block, SNN-based transformer blocks, and an ANN-based predictor head. First, to simplify the transformer block while preserving its capacity to aggregate local and global spatio-temporal features, we design a parallel spike transformer block to replace sequential sub-blocks. Additionally, we propose a simplified spiking self-attention mechanism that omits the value parameter without compromising the model's performance. Experiments conducted on four datasets-PURE, UBFC-rPPG, UBFC-Phys, and MMPD demonstrate that the proposed model achieves a 12.4\% reduction in power consumption compared to PhysFormer. Additionally, the power consumption of the transformer block is reduced by a factor of 12.2, while maintaining decent performance as PhysFormer and other ANN-based models.