zkLLM: Zero Knowledge Proofs for Large Language Models

📄 arXiv: 2404.16109v1 📥 PDF

作者: Haochen Sun, Jason Li, Hongyang Zhang

分类: cs.LG, cs.CR

发布日期: 2024-04-24

备注: Accepted to ACM CCS 2024, camera-ready version under preparation. This is the author's version of the work. It is posted here for your personal use. Not for redistribution

💡 一句话要点

提出zkLLM以解决大语言模型输出真实性验证问题

🎯 匹配领域: 支柱九：具身大模型 (Embodied Foundation Models)

关键词: 零知识证明 大语言模型 深度学习 注意力机制 CUDA实现 模型隐私 合法性验证

📋 核心要点

  1. 当前大语言模型的输出真实性验证面临法律挑战，现有方法无法有效解决这一问题。
  2. 本文提出zkLLM，结合tlookup和zkAttn，专门针对LLMs的非算术操作和注意力机制进行零知识证明设计。
  3. 实验表明，zkLLM在处理130亿参数的模型时，能够在15分钟内生成小于200 kB的证明，显著提升了验证效率。

📝 摘要（中文）

随着大语言模型（LLMs）的广泛应用，关于其合法性的问题日益突出，尤其是如何验证其输出的真实性。本文提出zkLLM，这是首个专为LLMs设计的零知识证明方案。我们引入tlookup，解决深度学习中非算术操作的挑战，并基于此开发了zkAttn，专门针对注意力机制进行优化。通过完全并行化的CUDA实现，zkLLM能够在15分钟内为拥有130亿参数的LLMs生成小于200 kB的正确性证明，确保模型参数的隐私性。

🔬 方法详解

问题定义：本文旨在解决大语言模型输出的真实性验证问题，现有方法在处理非算术操作时存在效率低下和隐私泄露的风险。

核心思路：提出zkLLM，利用tlookup和zkAttn实现高效的零知识证明，确保在验证过程中不泄露模型参数信息。

技术框架：zkLLM的整体架构包括tlookup模块用于处理非算术张量操作，以及zkAttn模块专门针对注意力机制进行优化，二者结合实现高效的零知识证明。

关键创新：zkLLM是首个专为大语言模型设计的零知识证明方案，尤其在处理非算术操作时无渐进开销，显著提升了验证效率。

关键设计：采用完全并行化的CUDA实现，优化了运行时间和内存使用，确保生成的证明小于200 kB，保护模型参数的隐私性。

📊 实验亮点

实验结果显示，zkLLM能够在15分钟内为130亿参数的LLMs生成正确性证明，证明大小小于200 kB，显著提升了验证效率，相较于传统方法，运行时间和内存使用均得到了优化。

🎯 应用场景

zkLLM的提出为大语言模型的合法性验证提供了新的解决方案，具有广泛的应用潜力。它可以被应用于法律合规、知识产权保护等领域，确保AI生成内容的可信性，推动AI技术的健康发展。未来，zkLLM可能会成为AI系统中不可或缺的组成部分，促进更广泛的应用场景。

📄 摘要（原文）

The recent surge in artificial intelligence (AI), characterized by the prominence of large language models (LLMs), has ushered in fundamental transformations across the globe. However, alongside these advancements, concerns surrounding the legitimacy of LLMs have grown, posing legal challenges to their extensive applications. Compounding these concerns, the parameters of LLMs are often treated as intellectual property, restricting direct investigations. In this study, we address a fundamental challenge within the realm of AI legislation: the need to establish the authenticity of outputs generated by LLMs. To tackle this issue, we present zkLLM, which stands as the inaugural specialized zero-knowledge proof tailored for LLMs to the best of our knowledge. Addressing the persistent challenge of non-arithmetic operations in deep learning, we introduce tlookup, a parallelized lookup argument designed for non-arithmetic tensor operations in deep learning, offering a solution with no asymptotic overhead. Furthermore, leveraging the foundation of tlookup, we introduce zkAttn, a specialized zero-knowledge proof crafted for the attention mechanism, carefully balancing considerations of running time, memory usage, and accuracy. Empowered by our fully parallelized CUDA implementation, zkLLM emerges as a significant stride towards achieving efficient zero-knowledge verifiable computations over LLMs. Remarkably, for LLMs boasting 13 billion parameters, our approach enables the generation of a correctness proof for the entire inference process in under 15 minutes. The resulting proof, compactly sized at less than 200 kB, is designed to uphold the privacy of the model parameters, ensuring no inadvertent information leakage.