Beyond FLOPs: Energy-Aware Knowledge Distillation for Sustainable LLMs on Code-Related Task

📄 arXiv: 2608.17515v1 📥 PDF

作者: Enrique Barba Roque, Luís Cruz, Annibale Panichella

分类: cs.SE, cs.AI

发布日期: 2026-08-18

💡 一句话要点

提出能效知识蒸馏以提升代码相关任务的LLMs可持续性

🎯 匹配领域: 支柱二：RL算法与架构 (RL & Architecture) 支柱九：具身大模型 (Embodied Foundation Models)

关键词: 大型语言模型 知识蒸馏 能效优化 软件工程 能量代理模型 可持续性 代码相关任务

📋 核心要点

  1. 现有大型语言模型在软件工程任务中能耗高，限制了其在消费硬件和资源受限平台的应用。
  2. 论文提出了一种基于能量代理的知识蒸馏方法，旨在提高模型效率并验证FLOPs作为能耗指标的有效性。
  3. 实验结果显示，使用能量代理模型的蒸馏模型在推理能耗和内存使用上显著降低，同时保持了较高的准确性。

📝 摘要（中文）

背景：大型语言模型（LLMs）在软件工程任务中应用广泛，但其高计算需求和能耗引发可持续性问题。目的：本论文探讨能效知识蒸馏在软件工程中的应用，旨在提高模型效率并验证FLOPs作为能耗指标的可靠性。方法：通过Morph方法进行控制实验，研究FLOPs与能耗的关系，并引入能量代理模型来估算CPU和GPU能耗。结果：研究表明FLOPs并非总是能耗的可靠指标，使用能量代理模型可获得更佳效果。蒸馏后的学生模型在推理时能耗降低高达90%，内存使用减少86%，仅有适度的准确性折衷。结论：基于直接能量代理的能效知识蒸馏可提升LLMs在软件工程应用中的能耗、可持续性和可部署性。

🔬 方法详解

问题定义：本论文旨在解决大型语言模型在软件工程任务中高能耗的问题。现有方法主要依赖FLOPs作为能耗指标，但其可靠性存疑。

核心思路：论文提出通过能量代理模型来指导知识蒸馏过程，从而提高模型的能效和可持续性。这样的设计能够更准确地反映模型在实际应用中的能耗。

技术框架：整体架构包括Morph蒸馏方法和能量代理模型。Morph用于优化蒸馏过程，而能量代理模型则在优化过程中实时估算CPU和GPU的能耗。

关键创新：最重要的创新在于引入能量代理模型，替代传统的FLOPs指标，使得能耗评估更加准确。与现有方法相比，这一创新显著提升了模型的能效。

关键设计：在实验中，设置了多个参数以优化蒸馏过程，并设计了特定的损失函数来平衡能耗与模型性能之间的折衷。

🖼️ 关键图片

img_0
img_1
img_2

📊 实验亮点

实验结果显示，使用能量代理模型的蒸馏学生模型在推理能耗上降低了高达90%，内存使用减少了86%，仅有适度的准确性折衷。这些结果表明，能效知识蒸馏在提升模型性能的同时，显著改善了能耗表现。

🎯 应用场景

该研究的潜在应用领域包括软件工程中的代码克隆检测、漏洞预测和代码摘要等任务。通过提高大型语言模型的能效和可持续性，研究成果将推动其在资源受限环境中的应用，具有重要的实际价值和未来影响。

📄 摘要（原文）

Background: Large Language Models (LLMs) are increasingly being applied to Software Engineering (SE) tasks, achieving high accuracy across problems such as clone detection, vulnerability prediction, and code summarization. However, their high computational demands and energy consumption raise sustainability concerns and hinder their use on consumer hardware and resource-constrained platforms. A common way to report the computational cost of an LLM in the literature and industry is to use the number of Floating Point Operations (FLOPs) required to perform a pass over the network. Aims: This paper investigates the implications of energy-aware knowledge distillation for SE, aiming to improve model efficiency while maintaining performance and to determine whether FLOPs is a reliable energy-aware metric. Method: We conduct a controlled experiment using Morph, a Many-Objective Optimization-based distillation methodology, to empirically examine whether FLOPs accurately reflect energy consumption in Clone Detection and Vulnerability Prediction tasks. We extend this methodology to include energy-surrogate models that directly estimate CPU and GPU energy consumption during optimization, and we apply Morph to generative tasks using CodeT5+ for code summarization. Results: Our results show that FLOPs is not always a reliable indicator of energy consumption, and better results can be achieved by using energy-surrogate models. Distilled student models can reduce inference energy consumption by up to 90\% and memory usage by 86\%, with only modest accuracy trade-offs. Conclusions: Energy-aware knowledge distillation when guided by direct energy surrogates rather than FLOPs can improve the energy consumption, sustainability, and deployability of LLMs for SE applications, enabling efficient models on consumer hardware.