Pruning random resistive memory for optimizing analogue AI

📄 arXiv: 2311.07164v1 📥 PDF

作者: Yi Li, Songqi Wang, Yaping Zhao, Shaocong Wang, Woyu Zhang, Yangu He, Ning Lin, Binbin Cui, Xi Chen, Shiming Zhang, Hao Jiang, Peng Lin, Xumeng Zhang, Xiaojuan Qi, Zhongrui Wang, Xiaoxin Xu, Dashan Shang, Qi Liu, Kwang-Ting Cheng, Ming Liu

分类: cs.ET, cs.AI, cs.AR

发布日期: 2023-11-13

💡 一句话要点

提出软件-硬件协同设计以优化模拟AI中的随机电阻存储器

🎯 匹配领域: 支柱九：具身大模型 (Embodied Foundation Models)

关键词: 模拟计算 电阻存储器 神经网络 边缘修剪 能效提升 人工智能硬件 软件-硬件协同设计

📋 核心要点

  1. 现有的模拟计算方法面临编程非理想性和高成本的问题，限制了其在人工智能硬件中的应用。
  2. 本文提出通过软件-硬件协同设计，利用边缘修剪优化随机加权神经网络的拓扑结构，而非精确调节电阻存储器权重。
  3. 在实验中，采用40nm电阻存储器宏实现了显著的性能提升，分类准确率和能效均有大幅改善。

📝 摘要（中文）

人工智能的快速发展伴随着大型语言模型的出现，这些模型在智能表现上接近人类。然而，这些模型在能耗和环境可持续性方面面临前所未有的挑战。本文提出了一种通用解决方案，通过结构性可塑性启发的边缘修剪技术，优化随机加权的模拟电阻存储器神经网络的拓扑结构。我们在40nm 256K电阻存储器宏上实现了这一协同设计，在FashionMNIST和Spoken digits数据集上分别观察到17.3%和19.9%的分类准确率提升，同时在DRIVE数据集的图像分割中，PR（ROC）分别提高了9.8%和2%。得益于模拟内存计算，能效提升分别达到82.1%、51.2%和99.8%。

🔬 方法详解

问题定义：本文旨在解决模拟计算中编程非理想性和高成本的问题，这些问题限制了电阻存储器在人工智能中的应用。

核心思路：通过软件-硬件协同设计，采用结构性可塑性启发的边缘修剪技术，优化随机加权神经网络的拓扑结构，避免了对电阻存储器权重的精确调节。

技术框架：整体架构包括两个主要模块：软件模块负责优化神经网络的连接拓扑，硬件模块利用透射电子显微镜揭示编程随机性的物理来源，以实现大规模低成本的实现。

关键创新：最重要的创新在于结合了软件优化与硬件特性，利用随机性来提升模拟计算的性能，这与传统方法的精确调节形成鲜明对比。

关键设计：在设计中，采用了特定的损失函数和网络结构，以支持边缘修剪的有效实施，并确保高性能子网络的形成。通过这些设计，能够在保持高效能的同时，降低能耗。

🖼️ 关键图片

fig_0
fig_1
fig_2

📊 实验亮点

实验结果显示，在FashionMNIST和Spoken digits数据集上，分类准确率分别提高了17.3%和19.9%；在DRIVE数据集的图像分割中，PR（ROC）分别提升了9.8%和2%。此外，能效提升分别达到82.1%、51.2%和99.8%，显示出模拟内存计算的巨大潜力。

🎯 应用场景

该研究的潜在应用领域包括智能硬件、边缘计算和物联网设备等，能够显著提升这些领域中人工智能系统的能效和性能。未来，随着模拟计算技术的进一步发展，可能会在更广泛的AI应用中发挥重要作用。

📄 摘要（原文）

The rapid advancement of artificial intelligence (AI) has been marked by the large language models exhibiting human-like intelligence. However, these models also present unprecedented challenges to energy consumption and environmental sustainability. One promising solution is to revisit analogue computing, a technique that predates digital computing and exploits emerging analogue electronic devices, such as resistive memory, which features in-memory computing, high scalability, and nonvolatility. However, analogue computing still faces the same challenges as before: programming nonidealities and expensive programming due to the underlying devices physics. Here, we report a universal solution, software-hardware co-design using structural plasticity-inspired edge pruning to optimize the topology of a randomly weighted analogue resistive memory neural network. Software-wise, the topology of a randomly weighted neural network is optimized by pruning connections rather than precisely tuning resistive memory weights. Hardware-wise, we reveal the physical origin of the programming stochasticity using transmission electron microscopy, which is leveraged for large-scale and low-cost implementation of an overparameterized random neural network containing high-performance sub-networks. We implemented the co-design on a 40nm 256K resistive memory macro, observing 17.3% and 19.9% accuracy improvements in image and audio classification on FashionMNIST and Spoken digits datasets, as well as 9.8% (2%) improvement in PR (ROC) in image segmentation on DRIVE datasets, respectively. This is accompanied by 82.1%, 51.2%, and 99.8% improvement in energy efficiency thanks to analogue in-memory computing. By embracing the intrinsic stochasticity and in-memory computing, this work may solve the biggest obstacle of analogue computing systems and thus unleash their immense potential for next-generation AI hardware.