MK-SGN: A Spiking Graph Convolutional Network with Multimodal Fusion and Knowledge Distillation for Skeleton-based Action Recognition

📄 arXiv: 2404.10210v5 📥 PDF

作者: Naichuan Zheng, Hailun Xia, Zeyu Liang, Yuchen Du

分类: cs.CV

发布日期: 2024-04-16 (更新: 2025-11-19)

DOI: 10.1016/j.neucom.2025.131796

💡 一句话要点

提出MK-SGN以解决骨架动作识别中的能耗问题

🎯 匹配领域: 支柱二：RL算法与架构 (RL & Architecture) 支柱七：动作重定向 (Motion Retargeting) 支柱九：具身大模型 (Embodied Foundation Models)

关键词: 骨架动作识别 脉冲神经网络 图卷积网络 多模态融合 知识蒸馏 能效优化 深度学习

📋 核心要点

  1. 现有的基于GCN的动作识别方法在能耗方面存在显著挑战，难以在边缘设备上有效部署。
  2. MK-SGN通过结合脉冲神经网络的能效与图卷积网络的表示能力，提出了一种新颖的多模态融合和知识蒸馏策略。
  3. 实验结果表明，MK-SGN在能耗上减少超过98%，同时在识别准确率上超越了现有的最先进方法。

📝 摘要（中文）

近年来，多模态图卷积网络（GCNs）在基于骨架的动作识别中取得了显著的性能。然而，GCN方法中固有的高能耗连续浮点运算在能源受限的边缘设备上部署时面临重大挑战。为了解决这些局限性，本文首次提出了MK-SGN，一种结合多模态融合和知识蒸馏的脉冲图卷积网络，利用脉冲神经网络（SNNs）的能效进行骨架动作识别。通过将SNN的节能特性与GCN的图表示能力相结合，MK-SGN在保持竞争性识别准确率的同时显著降低了能耗。该方法在能效和识别准确率上均超越了现有的最先进框架，展示了在骨架动作识别领域的广泛应用潜力。

🔬 方法详解

问题定义：本研究旨在解决基于骨架的动作识别中，现有GCN方法的高能耗问题，限制了其在边缘设备上的应用。

核心思路：MK-SGN结合了脉冲神经网络的能效与图卷积网络的图表示能力，通过多模态融合和知识蒸馏来提升性能和降低能耗。

技术框架：MK-SGN的整体架构包括三个主要模块：脉冲多模态融合（SMF）模块、基于自注意力的脉冲图卷积（SA-SGC）模块和脉冲时间卷积（STC）模块，分别用于融合多模态数据、捕捉空间关系和时间动态。

关键创新：该研究的核心创新在于首次将脉冲神经网络应用于骨架动作识别，并通过知识蒸馏策略有效提升了脉冲图卷积网络的性能。

关键设计：在设计中，采用了中间层蒸馏和软标签蒸馏的综合知识蒸馏策略，确保了信息从多模态GCN有效转移到SGN，提升了模型的整体表现。

🖼️ 关键图片

fig_0
fig_1
fig_2

📊 实验亮点

实验结果显示，MK-SGN在能耗方面相比传统GCN方法减少超过98%，同时在识别准确率上超越了现有的最先进SNN框架，展示了其在能效与性能上的双重优势。

🎯 应用场景

MK-SGN的研究成果在智能监控、虚拟现实、智能家居等领域具有广泛的应用潜力。其高能效特性使得在电池供电的设备上实现实时动作识别成为可能，推动了边缘计算技术的发展。未来，该方法可为更多低功耗、高性能的智能设备提供支持。

📄 摘要（原文）

In recent years, multimodal Graph Convolutional Networks (GCNs) have achieved remarkable performance in skeleton-based action recognition. The reliance on high-energy-consuming continuous floating-point operations inherent in GCN-based methods poses significant challenges for deployment in energy-constrained, battery-powered edge devices. To address these limitations, MK-SGN, a Spiking Graph Convolutional Network with Multimodal Fusion and Knowledge Distillation, is proposed to leverage the energy efficiency of Spiking Neural Networks (SNNs) for skeleton-based action recognition for the first time. By integrating the energy-saving properties of SNNs with the graph representation capabilities of GCNs, MK-SGN achieves significant reductions in energy consumption while maintaining competitive recognition accuracy. Firstly, we formulate a Spiking Multimodal Fusion (SMF) module to effectively fuse multimodal skeleton data represented as spike-form features. Secondly, we propose the Self-Attention Spiking Graph Convolution (SA-SGC) module and the Spiking Temporal Convolution (STC) module, to capture spatial relationships and temporal dynamics of spike-form features. Finally, we propose an integrated knowledge distillation strategy to transfer information from the multimodal GCN to the SGN, incorporating both intermediate-layer distillation and soft-label distillation to enhance the performance of the SGN. MK-SGN exhibits substantial advantages, surpassing state-of-the-art GCN frameworks in energy efficiency and outperforming state-of-the-art SNN frameworks in recognition accuracy. The proposed method achieves a remarkable reduction in energy consumption, exceeding 98\% compared to conventional GCN-based approaches. This research establishes a robust baseline for developing high-performance, energy-efficient SNN-based models for skeleton-based action recognition