System-level Impact of Non-Ideal Program-Time of Charge Trap Flash (CTF) on Deep Neural Network
作者: S. Shrivastava, A. Biswas, S. Chakrabarty, G. Dash, V. Saraswat, U. Ganguly
分类: cs.NE, cs.AI, cs.ET, eess.IV
发布日期: 2024-02-15
💡 一句话要点
提出脉冲列设计补偿技术以解决CTF非理想编程时间问题
🎯 匹配领域: 支柱八：物理动画 (Physics-based Animation)
关键词: 电荷陷阱闪存 深度神经网络 电阻处理单元 脉冲列设计 能效优化 随机计算 权重更新 训练性能
📋 核心要点
- 现有方法未能充分考虑CTF非理想编程时间对深度神经网络学习性能的影响，导致训练效果不佳。
- 论文提出了一种脉冲列设计补偿技术，通过优化输入脉冲数量和脉冲间隔来减少误差，提高学习性能。
- 实验结果表明，当输入脉冲数量较大时，学习性能接近理想水平，而较小的输入脉冲数量则对脉冲间隔敏感。
📝 摘要（中文）
深度神经网络（DNN）利用电阻处理单元（RPU）架构进行学习，具有高能效，能够实现基于存储计算的权重更新。尽管电荷陷阱闪存（CTF）设备能够支持RPU的权重更新，但现有研究表明，CTF的非理想编程时间会影响权重更新的效果。本研究提出了一种脉冲列设计补偿技术，旨在减少因CTF非理想编程时间和网络随机性造成的总误差。通过在MNIST和Fashion-MNIST数据集上进行模拟，发现较大的输入脉冲数量（N）能使学习性能接近理想水平，而较小的N则对脉冲间隔（t_gap）敏感。最终的消融研究表明，权重更新中的低噪声水平是提升学习性能的关键因素。
🔬 方法详解
问题定义：本研究旨在解决电荷陷阱闪存（CTF）在深度神经网络（DNN）训练中，由于非理想编程时间导致的权重更新不准确的问题。现有方法未能有效应对这一挑战，影响了训练效果。
核心思路：论文提出了一种脉冲列设计补偿技术，旨在通过优化输入脉冲数量（N）和脉冲间隔（t_gap）来降低因非理想编程时间和随机性引起的误差，从而提升DNN的学习性能。
技术框架：整体架构包括两个主要模块：首先是脉冲列设计补偿模块，通过调整脉冲参数来减少误差；其次是基于RPU的DNN训练模块，使用MNIST和Fashion-MNIST数据集进行模拟实验。
关键创新：本研究的主要创新在于提出了脉冲列设计补偿技术，能够有效降低CTF非理想编程时间对学习性能的影响，这一方法与传统的权重更新方法有本质区别。
关键设计：在实验中，输入脉冲数量（N）设置为1000时，学习性能接近理想水平，而当N小于500时，学习性能显著依赖于脉冲间隔（t_gap）。
📊 实验亮点
实验结果显示，当输入脉冲数量N达到约1000时，学习性能接近理想水平，且对脉冲间隔t_gap的选择影响较小；而在N小于500时，学习性能显著依赖于t_gap的设置，表明脉冲设计对训练效果的重要性。
🎯 应用场景
该研究的潜在应用领域包括智能硬件、边缘计算和嵌入式系统等，能够为基于RPU的深度学习系统提供更高的能效和准确性，推动神经网络在实际应用中的发展。
📄 摘要（原文）
Learning of deep neural networks (DNN) using Resistive Processing Unit (RPU) architecture is energy-efficient as it utilizes dedicated neuromorphic hardware and stochastic computation of weight updates for in-memory computing. Charge Trap Flash (CTF) devices can implement RPU-based weight updates in DNNs. However, prior work has shown that the weight updates (V_T) in CTF-based RPU are impacted by the non-ideal program time of CTF. The non-ideal program time is affected by two factors of CTF. Firstly, the effects of the number of input pulses (N) or pulse width (pw), and secondly, the gap between successive update pulses (t_gap) used for the stochastic computation of weight updates. Therefore, the impact of this non-ideal program time must be studied for neural network training simulations. In this study, Firstly, we propose a pulse-train design compensation technique to reduce the total error caused by non-ideal program time of CTF and stochastic variance of a network. Secondly, we simulate RPU-based DNN with non-ideal program time of CTF on MNIST and Fashion-MNIST datasets. We find that for larger N (~1000), learning performance approaches the ideal (software-level) training level and, therefore, is not much impacted by the choice of t_gap used to implement RPU-based weight updates. However, for lower N (<500), learning performance depends on T_gap of the pulses. Finally, we also performed an ablation study to isolate the causal factor of the improved learning performance. We conclude that the lower noise level in the weight updates is the most likely significant factor to improve the learning performance of DNN. Thus, our study attempts to compensate for the error caused by non-ideal program time and standardize the pulse length (N) and pulse gap (t_gap) specifications for CTF-based RPUs for accurate system-level on-chip training.