MELTing point: Mobile Evaluation of Language Transformers

📄 arXiv: 2403.12844v4 📥 PDF

作者: Stefanos Laskaridis, Kleomenis Katevas, Lorenzo Minto, Hamed Haddadi

分类: cs.LG

发布日期: 2024-03-19 (更新: 2024-07-25)

备注: Accepted at the 30th Annual International Conference On Mobile Computing And Networking (MobiCom 2024)

💡 一句话要点

提出MELT框架以解决移动设备上大语言模型执行效率问题

🎯 匹配领域: 支柱九：具身大模型 (Embodied Foundation Models)

关键词: 大语言模型 移动设备 执行效率 性能评估 量化技术 能效分析 MELT框架

📋 核心要点

  1. 现有大语言模型在移动设备上的执行效率低，内存和能耗限制了其广泛应用。
  2. 提出MELT框架，支持多种设备和模型的无头执行与基准测试，量化性能和能效。
  3. 实验结果显示，LLM推理主要受内存限制，量化技术虽降低内存需求，但准确性有所下降。

📝 摘要（中文）

随着变换器（Transformers）在机器学习领域的革命性进展，它们逐渐被应用于日常任务中，赋予计算机“智能火花”。然而，其运行时需求限制了在移动设备上的广泛部署。本文探讨了大语言模型（LLMs）在移动设备上的执行现状，提出了MELT自动化基础设施，支持在不同设备和框架上进行无头执行和基准测试。通过评估流行的指令微调LLMs，量化了性能、能效和准确性，展示了超大规模模型时代的设备智能现状。结果显示，LLM推理主要受内存限制，量化显著降低内存需求但会影响准确性，且持续执行LLMs在能耗和热行为方面仍面临挑战。

🔬 方法详解

问题定义：本文旨在解决大语言模型在移动设备上执行效率低的问题，现有方法在内存和能耗方面存在显著限制，导致无法广泛应用于个人设备。

核心思路：提出MELT框架，通过支持无头执行和基准测试，评估不同模型在移动设备上的性能，旨在提升执行效率和用户体验。

技术框架：MELT框架包括模型加载、执行、性能监测和结果分析等模块，支持Android、iOS及Nvidia Jetson等多种设备，能够进行全面的性能评估。

关键创新：首次系统性研究了移动设备上大语言模型的执行，量化了性能、能效和准确性，揭示了内存限制对推理的影响。

关键设计：在实验中采用了不同的量化策略，设置了多种参数以优化内存使用，同时监测能耗和热行为，以确保在移动设备上可行的执行方案。

🖼️ 关键图片

fig_0
fig_1
fig_2

📊 实验亮点

实验结果表明，LLM推理在不同设备上的性能差异显著，量化技术能够将内存需求降低至原来的30%，但准确性下降约5%。此外，持续执行LLMs的能耗和热行为问题仍需解决，以改善用户体验。

🎯 应用场景

该研究的潜在应用领域包括智能手机、平板电脑及边缘计算设备等，能够为移动设备上的自然语言处理任务提供高效的解决方案。随着个人设备性能的提升，MELT框架有望推动大语言模型在移动端的广泛应用，提升用户体验和隐私保护。

📄 摘要（原文）

Transformers have revolutionized the machine learning landscape, gradually making their way into everyday tasks and equipping our computers with "sparks of intelligence". However, their runtime requirements have prevented them from being broadly deployed on mobile. As personal devices become increasingly powerful and prompt privacy becomes an ever more pressing issue, we explore the current state of mobile execution of Large Language Models (LLMs). To achieve this, we have created our own automation infrastructure, MELT, which supports the headless execution and benchmarking of LLMs on device, supporting different models, devices and frameworks, including Android, iOS and Nvidia Jetson devices. We evaluate popular instruction fine-tuned LLMs and leverage different frameworks to measure their end-to-end and granular performance, tracing their memory and energy requirements along the way. Our analysis is the first systematic study of on-device LLM execution, quantifying performance, energy efficiency and accuracy across various state-of-the-art models and showcases the state of on-device intelligence in the era of hyperscale models. Results highlight the performance heterogeneity across targets and corroborates that LLM inference is largely memory-bound. Quantization drastically reduces memory requirements and renders execution viable, but at a non-negligible accuracy cost. Drawing from its energy footprint and thermal behavior, the continuous execution of LLMs remains elusive, as both factors negatively affect user experience. Last, our experience shows that the ecosystem is still in its infancy, and algorithmic as well as hardware breakthroughs can significantly shift the execution cost. We expect NPU acceleration, and framework-hardware co-design to be the biggest bet towards efficient standalone execution, with the alternative of offloading tailored towards edge deployments.