SOEN-101: Code Generation by Emulating Software Process Models Using Large Language Model Agents
作者: Feng Lin, Dong Jae Kim, Tse-Husn, Chen
分类: cs.SE, cs.AI
发布日期: 2024-03-23 (更新: 2024-10-31)
备注: ICSE 2025
💡 一句话要点
提出FlowGen框架以提升代码生成质量
🎯 匹配领域: 支柱九：具身大模型 (Embodied Foundation Models)
关键词: 代码生成 大型语言模型 软件过程模型 协作机制 性能提升
📋 核心要点
- 现有代码生成方法缺乏有效的协作机制，导致生成代码的质量和稳定性不足。
- 本文提出FlowGen框架，通过模拟软件过程模型，利用多个LLM代理协作生成代码，提升代码质量。
- 实验结果显示，FlowGenScrum在多个基准测试中表现优异，较现有方法有显著提升，尤其在MBPP基准上表现突出。
📝 摘要（中文）
软件过程模型在软件团队协作和沟通中至关重要，能够帮助解决复杂的开发任务。本文提出了FlowGen，一个基于多个大型语言模型（LLM）代理的代码生成框架，模拟软件过程模型。我们通过分配LLM代理扮演需求工程师、架构师、开发者、测试者和敏捷教练等角色，组织其沟通模式，来模拟三种过程模型：FlowGenWaterfall、FlowGenTDD和FlowGenScrum。实验结果表明，FlowGenScrum在多个基准测试中表现优异，Pass@1分别为75.2、65.5、82.5和56.7，较RawGPT平均提升15%。此外，FlowGenScrum在MBPP中超越了CodeT，且与CodeT的结合显著提高了性能，显示了软件过程模型在提升LLM生成代码质量和稳定性方面的有效性。
🔬 方法详解
问题定义：现有的代码生成方法往往缺乏有效的团队协作和沟通机制，导致生成的代码质量不高且不稳定。
核心思路：本文提出FlowGen框架，通过模拟软件过程模型，利用多个大型语言模型（LLM）代理来扮演不同的角色，协同生成高质量代码。
技术框架：FlowGen框架包括多个LLM代理，每个代理负责特定的角色（如需求工程师、架构师等），并通过链式思维和提示组合进行协作。框架分为三个主要模块：角色分配、沟通模式组织和自我优化。
关键创新：最重要的创新在于通过模拟软件过程模型来组织LLM代理的协作，显著提升了代码生成的质量和稳定性。这种方法与传统的单一模型生成方法本质上不同。
关键设计：在设计中，使用GPT3.5作为基础LLM，设置了多个基准测试（如HumanEval、MBPP等）来评估生成效果，并通过参数调整和模型集成（如与CodeT结合）实现性能提升。具体的温度值和版本设置保持稳定，确保结果的可靠性。
🖼️ 关键图片
📊 实验亮点
实验结果显示，FlowGenScrum在HumanEval、HumanEval-ET、MBPP和MBPP-ET基准测试中的Pass@1分别为75.2、65.5、82.5和56.7，较RawGPT平均提升15%。此外，FlowGenScrum在MBPP中超越了CodeT，且与CodeT的结合显著提高了性能，达到了最高的Pass@1分数。
🎯 应用场景
该研究的潜在应用领域包括软件开发工具、自动化测试生成和智能编程助手等。通过提升代码生成的质量和稳定性，FlowGen框架能够显著提高软件开发效率，降低开发成本，未来可能在软件工程领域产生深远影响。
📄 摘要（原文）
Software process models are essential to facilitate collaboration and communication among software teams to solve complex development tasks. Inspired by these software engineering practices, we present FlowGen - a code generation framework that emulates software process models based on multiple Large Language Model (LLM) agents. We emulate three process models, FlowGenWaterfall, FlowGenTDD, and FlowGenScrum, by assigning LLM agents to embody roles (i.e., requirement engineer, architect, developer, tester, and scrum master) that correspond to everyday development activities and organize their communication patterns. The agents work collaboratively using chain-of-thought and prompt composition with continuous self-refinement to improve the code quality. We use GPT3.5 as our underlying LLM and several baselines (RawGPT, CodeT, Reflexion) to evaluate code generation on four benchmarks: HumanEval, HumanEval-ET, MBPP, and MBPP-ET. Our findings show that FlowGenScrum excels compared to other process models, achieving a Pass@1 of 75.2, 65.5, 82.5, and 56.7 in HumanEval, HumanEval-ET, MBPP, and MBPP-ET, respectively (an average of 15% improvement over RawGPT). Compared with other state-of-the-art techniques, FlowGenScrum achieves a higher Pass@1 in MBPP compared to CodeT, with both outperforming Reflexion. Notably, integrating CodeT into FlowGenScrum resulted in statistically significant improvements, achieving the highest Pass@1 scores. Our analysis also reveals that the development activities impacted code smell and exception handling differently, with design and code review adding more exception handling and reducing code smells. Finally, FlowGen models maintain stable Pass@1 scores across GPT3.5 versions and temperature values, highlighting the effectiveness of software process models in enhancing the quality and stability of LLM-generated code.