DeepCode AI Fix: Fixing Security Vulnerabilities with Large Language Models
作者: Berkay Berabi, Alexey Gronskiy, Veselin Raychev, Gishor Sivanrupan, Victor Chibotaru, Martin Vechev
分类: cs.CR, cs.LG, cs.PL, cs.SE
发布日期: 2024-02-19 (更新: 2024-02-23)
备注: 26 pages, 13 figures (v2, small fix in author affiliations)
💡 一句话要点
提出深度代码AI修复方法以解决安全漏洞问题
🎯 匹配领域: 支柱九：具身大模型 (Embodied Foundation Models)
关键词: 自动程序修复 安全漏洞 大型语言模型 程序分析 代码修复 深度学习 机器学习
📋 核心要点
- 现有的自动程序修复方法在处理复杂的语义错误（如安全漏洞）时效果不佳，主要由于缺乏足够的训练数据和长距离代码关系的学习能力。
- 本文提出了一种新颖的查询和微调LLM的方法，通过程序分析来限制模型的注意力机制，从而专注于修复所需的代码片段。
- 实验结果显示，使用该代码缩减方法，Mixtral-8x7B模型能够去除超过80%的缺陷，并在10%至50%的案例中与人工修复完全一致，超越了GPT-3.5和GPT-4等基线模型。
📝 摘要（中文）
自动程序修复领域近年来受到广泛关注，但针对复杂语义错误（如安全漏洞）的有效修复系统仍然难以实现。本文探讨了利用大型语言模型（LLMs）解决代码修复任务的有效性。我们提出了一种新方法，通过程序分析限制LLM的注意力机制，仅关注修复所需的代码部分，从而显著减少训练数据的需求。实验结果表明，该方法在修复准确性和效率上均有显著提升，能够有效去除超过80%的报告缺陷，并在10%至50%的情况下与人工修复完全匹配。
🔬 方法详解
问题定义：本文旨在解决自动程序修复中针对复杂语义错误（如安全漏洞）的有效性问题。现有方法在处理此类问题时，往往面临训练数据不足和长距离代码关系学习困难的挑战。
核心思路：论文提出通过程序分析来优化LLM的注意力机制，使其仅关注修复所需的代码片段，从而减少对大量训练数据的依赖。这种方法旨在提高模型在复杂代码修复任务中的表现。
技术框架：整体架构包括数据收集、程序分析、LLM微调和评估四个主要模块。首先，构建一个包含156种错误模式的综合数据集，然后通过程序分析提取相关代码片段，最后对LLM进行微调和评估。
关键创新：最重要的技术创新在于通过程序分析限制LLM的注意力机制，这与传统方法依赖于整个程序输入的方式有本质区别。该创新显著提高了模型在复杂修复任务中的表现。
关键设计：在模型训练中，采用了针对特定任务的损失函数，并设计了适应性学习率以优化微调过程。此外，模型结构上进行了调整，以更好地处理长距离依赖关系。具体参数设置和网络结构细节在论文中进行了详细描述。
🖼️ 关键图片
📊 实验亮点
实验结果显示，Mixtral-8x7B模型在修复任务中能够去除超过80%的报告缺陷，并在10%至50%的案例中与人工修复完全一致。该方法在性能上超越了基于GPT-3.5和GPT-4的基线模型，展示了显著的提升幅度。
🎯 应用场景
该研究的潜在应用领域包括软件开发、网络安全和自动化测试等。通过有效修复安全漏洞，能够显著提高软件的安全性和可靠性，减少人工干预的需求，提升开发效率。未来，该方法有望在更广泛的编程任务中得到应用，推动自动化程序修复技术的发展。
📄 摘要（原文）
The automated program repair field has attracted substantial interest over the years, but despite significant research efforts, creating a system that works well for complex semantic bugs such as security vulnerabilities has proven difficult. A promising direction to solve this challenge is by leveraging large language models (LLMs), which are increasingly used to solve various programming tasks. In this paper, we investigate the effectiveness of LLMs for solving code-repair task. We show that the task is difficult as it requires the model to learn long-range code relationships, a task that inherently relies on extensive amounts of training data. At the same time, creating a large, clean dataset for complex program bugs and their corresponding fixes is non-trivial. We propose a technique to address these challenges with a new approach for querying and fine-tuning LLMs. The idea is to use program analysis to limit the LLM's attention mechanism on the portions of code needed to perform the fix, drastically reducing the amount of required training data. Concretely, for training and inference, rather than feeding the entire program to the LLM, we reduce its code to a much shorter snippet that contains the reported defect together with the necessary context - and use that instead. Our evaluation shows that this code reduction approach substantially improves available models such as GPT-4 using few-shot learning, as well as fine-tuning models. To train and evaluate our system, we created a comprehensive code fixing dataset by extensively labeling 156 bug patterns (including 40 security rules), requiring complex interprocedural dataflow to discover. Our best system with Mixtral-8x7B can remove more than 80% of the reported defects while exactly matching the human fix in between 10 and 50% of cases, outperforming baselines based on GPT-3.5 and GPT-4, or based on window-based models like TFix.