Homomorphic WiSARDs: Efficient Weightless Neural Network training over encrypted data

📄 arXiv: 2403.20190v1 📥 PDF

作者: Leonardo Neumann, Antonio Guimarães, Diego F. Aranha, Edson Borin

分类: cs.CR, cs.LG

发布日期: 2024-03-29

💡 一句话要点

提出同态WiSARD以解决加密数据上的神经网络训练问题

🎯 匹配领域: 支柱五：交互与反应 (Interaction & Reaction)

关键词: 同态加密 无权重神经网络 数据隐私 机器学习 加密数据训练 WiSARD 图像识别

📋 核心要点

  1. 现有的同态评估方法在加密数据上进行机器学习训练面临效率低下和准确率下降的挑战。
  2. 本文提出了WiSARD及其后续的无权重神经网络（WNN），实现了在加密数据上高效的训练和推理。
  3. 实验结果显示，MNIST数据集上训练速度提升至93.8%准确率，HAM10000数据集上速度提升至69.9%准确率，显著优于现有方法。

📝 摘要（中文）

随着机器学习算法的广泛应用，数据隐私问题日益受到关注。现有的同态评估方法能够直接在加密数据上执行操作，确保数据的机密性。然而，训练过程仍然面临挑战，现有解决方案往往依赖于轻量级算法，难以处理复杂问题。本文提出了Wilkie、Stonham和Aleksander的识别设备（WiSARD）及其后续的无权重神经网络（WNN）在加密数据上的同态评估，展示了相较于卷积神经网络（CNN）更优的性能和较小的准确率下降。我们开发了一个完整的框架，能够在MNIST数据集上实现91.7%的准确率，仅需3.5分钟的加密训练，3.5小时后可达93.8%。在HAM10000数据集上，1.5分钟内达到67.9%的准确率，1小时后可达69.9%。与现有的同态评估CNN训练方法Glyph相比，速度提升高达1200倍，准确率损失最多为5.4%。

🔬 方法详解

问题定义：本文旨在解决在加密数据上进行神经网络训练的效率和准确率问题。现有方法多依赖轻量级算法，难以应对复杂的图像识别任务。

核心思路：提出同态WiSARD和无权重神经网络（WNN），通过在加密数据上进行同态评估，保持数据隐私的同时提高训练效率和准确率。

技术框架：整体框架包括多个独立模块，涵盖数据加密、同态计算、模型训练和推理等阶段，支持多线程加速训练过程。

关键创新：最重要的创新在于将WiSARD的同态评估引入到无权重神经网络训练中，显著提高了训练速度和准确率，尤其是在处理复杂任务时。

关键设计：在训练过程中，采用了多线程技术和优化的损失函数设计，确保在内存使用低于200MB的情况下，快速完成大规模数据集的训练。具体参数设置和网络结构设计在论文中详细阐述。

📊 实验亮点

实验结果显示，在MNIST数据集上，经过3.5分钟的加密训练，准确率达到91.7%，3.5小时后提升至93.8%。在HAM10000数据集上，1.5分钟内达到67.9%的准确率，1小时后可达69.9%。与Glyph相比，训练速度提升高达1200倍，准确率损失仅为5.4%。

🎯 应用场景

该研究在数据隐私保护和机器学习领域具有广泛的应用潜力，尤其适用于医疗、金融等对数据安全性要求极高的行业。通过在加密数据上进行高效训练，能够推动智能算法在敏感数据处理中的应用，促进数据驱动决策的安全性和有效性。

📄 摘要（原文）

The widespread application of machine learning algorithms is a matter of increasing concern for the data privacy research community, and many have sought to develop privacy-preserving techniques for it. Among existing approaches, the homomorphic evaluation of ML algorithms stands out by performing operations directly over encrypted data, enabling strong guarantees of confidentiality. The homomorphic evaluation of inference algorithms is practical even for relatively deep Convolution Neural Networks (CNNs). However, training is still a major challenge, with current solutions often resorting to lightweight algorithms that can be unfit for solving more complex problems, such as image recognition. This work introduces the homomorphic evaluation of Wilkie, Stonham, and Aleksander's Recognition Device (WiSARD) and subsequent Weightless Neural Networks (WNNs) for training and inference on encrypted data. Compared to CNNs, WNNs offer better performance with a relatively small accuracy drop. We develop a complete framework for it, including several building blocks that can be of independent interest. Our framework achieves 91.7% accuracy on the MNIST dataset after only 3.5 minutes of encrypted training (multi-threaded), going up to 93.8% in 3.5 hours. For the HAM10000 dataset, we achieve 67.9% accuracy in just 1.5 minutes, going up to 69.9% after 1 hour. Compared to the state of the art on the HE evaluation of CNN training, Glyph (Lou et al., NeurIPS 2020), these results represent a speedup of up to 1200 times with an accuracy loss of at most 5.4%. For HAM10000, we even achieved a 0.65% accuracy improvement while being 60 times faster than Glyph. We also provide solutions for small-scale encrypted training. In a single thread on a desktop machine using less than 200MB of memory, we train over 1000 MNIST images in 12 minutes or over the entire Wisconsin Breast Cancer dataset in just 11 seconds.