KernelSight-LM: A Kernel-Level LLM Inference Simulator

📄 arXiv: 2606.28565 📥 PDF

作者: Xiteng Yao, Taeho Kim, Hengzhi Pei, Xinle Liu, Kyle Ulrich, Leonard Lausen, Ashish Khetan, Xiang Song, George Karypis, Martin Herbordt

分类: cs.PF, cs.AI, cs.AR

发布日期: 2026-07-05

💡 一句话要点

提出KernelSight-LM以解决LLM推理性能评估问题

🎯 匹配领域: 支柱九：具身大模型 (Embodied Foundation Models)

关键词: 大型语言模型 推理性能 GPU内核模拟 性能评估 机器学习

📋 核心要点

  1. 现有方法在评估LLM推理性能时，面临服务层与GPU内核执行耦合的问题，导致基准测试效率低下。
  2. KernelSight-LM通过细粒度模拟，分解服务步骤并结合屋顶线模型与通信模型，提供高效的推理性能预测。
  3. 实验结果显示，KernelSight-LM在不同模型家族中，预测误差显著低于传统基准，提升幅度达到7.3倍。

📝 摘要（中文）

随着大型语言模型（LLMs）进入生产服务，实践者需要快速评估不同硬件、模型和服务参数下的推理性能，以满足成本和延迟目标。然而，LLMs的端到端行为将服务层策略与低级GPU内核执行紧密耦合，导致部署特定的基准测试缓慢且难以推广。本文提出KernelSight-LM，一个细粒度的推理模拟器，能够建模令牌级执行并生成内核级延迟分解。该模型将每个服务步骤分解为屋顶线内核模型、通信模型和主机开销模型，通过离散事件调度器组合，捕捉前缀缓存和连续批处理等机制。KernelSight-LM提供两种预测层次，分别在目标GPU数据与准确性之间进行权衡。

🔬 方法详解

问题定义：本文旨在解决大型语言模型推理性能评估中的效率低下问题，现有方法难以适应快速变化的硬件架构和服务需求。

核心思路：KernelSight-LM通过细粒度的模拟，结合屋顶线内核模型、通信模型和主机开销模型，提供高效的推理性能预测，旨在减少对目标GPU数据的依赖。

技术框架：整体架构包括三个主要模块：屋顶线内核模型、通信模型和主机开销模型，利用离散事件调度器进行组合，支持前缀缓存和连续批处理等机制。

关键创新：最重要的创新在于引入了学习的效率项和两种预测层次，显著提高了在未见GPU上的内核延迟预测准确性。

关键设计：采用了模型无关的内核微基准测试，优化了参数设置，确保在目标GPU上进行一次微基准测试即可显著提高预测精度。

🖼️ 关键图片

fig_0
fig_1
fig_2

📊 实验亮点

实验结果表明，KernelSight-LM在交叉代层次的内核延迟预测误差为12.1%，相比屋顶线基线提升了1.8倍；在目标测量层次中，误差降低至3.8%，提升幅度达到7.3倍，显示出其在性能评估中的显著优势。

🎯 应用场景

KernelSight-LM的潜在应用领域包括云计算服务、边缘计算和高性能计算等场景，能够帮助开发者在不同硬件环境中快速评估和优化LLM的推理性能，降低成本并提高响应速度。未来，该研究可能推动LLM在更多实际应用中的广泛部署。

📄 摘要（原文）

As large language models (LLMs) move into production serving, practitioners must rapidly evaluate inference performance across diverse hardware, models, and serving parameters to meet cost and latency targets. However, the end-to-end behavior of LLMs couples serving-layer policies with low-level GPU kernel execution and rapidly evolving architectures, forcing slow, deployment-specific benchmarking that is hard to generalize.We present KernelSight-LM, a fine-grained inference simulator that models token-level execution and produces kernel-level latency breakdowns. It decomposes each serving step into a roofline kernel model with a learned efficiency term, a communication model, and a host-overhead model, composed through a discrete-event scheduler that also captures mechanisms like prefix caching and continuous batching.KernelSight-LM offers two prediction tiers that trade target-GPU data for accuracy. The cross-generation tier uses no target-GPU measurements, only hardware specifications and kernel microbenchmarks from previously profiled GPUs, and predicts per-kernel latency on an unseen GPU generation to 12.1% error, a 1.8x improvement over the roofline baseline (22.0%).A second target-measured tier adds one model-agnostic kernel-microbenchmark sweep on the target GPU, sharpening per-kernel error to 3.8%, a 7.3x improvement over a comparable baseline (27.7%). Both tiers require far less target-GPU data than the prior systems they extend.In our simulator, these predictions yield end-to-end median (p50) errors across six model families of 15.4%, 12.8%, and 3.0% (TTFT, TPOT, throughput) in the cross-generation tier and 14.3%, 6.2%, and 2.7% in the target-measured tier, matching dedicated profiling tools while collecting far less on-device data. Beyond prediction, its kernel-level bottleneck breakdowns support hardware/software co-design and capacity planning.