Insights from the Usage of the Ansible Lightspeed Code Completion Service

📄 arXiv: 2402.17442v4 📥 PDF

作者: Priyam Sahoo, Saurabh Pujar, Ganesh Nalawade, Richard Gebhardt, Louis Mandel, Luca Buratti

分类: cs.SE, cs.AI, cs.PL

发布日期: 2024-02-27 (更新: 2024-10-22)

备注: This paper has been published at the 39th IEEE/ACM International Conference on Automated Software Engineering (ASE 2024), Industry Showcase under the title "Ansible Lightspeed: A Code Generation Service for IT Automation"

💡 一句话要点

提出Ansible Lightspeed以提升IT自动化领域代码生成效率

🎯 匹配领域: 支柱九：具身大模型 (Embodied Foundation Models)

关键词: 大型语言模型 代码生成 IT自动化 Ansible 用户接受率 自然语言处理 开发者工具

📋 核心要点

  1. 现有的代码助手大多聚焦于通用编程语言，缺乏对特定领域语言的支持，导致IT自动化领域的开发效率低下。
  2. Ansible Lightspeed服务专为生成Ansible YAML而设计，利用自然语言提示生成代码，填补了特定领域工具的空白。
  3. 通过对10,696名用户的分析，研究首次展示了N-Day用户留存率为13.66%，并提出了强接受率的新指标，显示出该工具的有效性。

📝 摘要（中文）

随着大型语言模型（LLMs）的出现，开发者生产力工具得以快速发展。现有的代码助手大多集中于通用编程语言，而针对特定领域语言的工具相对较少。Ansible Lightspeed是一种专为生成Ansible YAML而设计的LLM服务，能够根据自然语言提示生成代码。本文介绍了Ansible Lightspeed的设计与实现，并通过对10,696名真实用户的评估，分析其对开发者的实用性，提出了改进的用户接受率指标。研究结果表明，Lightspeed在多行Ansible任务建议中实现了49.08%的强接受率，展示了专用模型在特定领域的有效性。

🔬 方法详解

问题定义：本文旨在解决现有代码助手对特定领域语言支持不足的问题，尤其是在IT自动化领域，导致开发者在使用Ansible时效率低下。

核心思路：Ansible Lightspeed通过自然语言处理生成Ansible YAML代码，专注于特定领域的需求，提供更精准的代码建议，从而提升开发者的工作效率。

技术框架：整体架构包括自然语言输入模块、LLM代码生成模块和用户反馈分析模块。用户通过自然语言输入需求，系统生成相应的Ansible代码，并根据用户的编辑反馈进行优化。

关键创新：本文提出的强接受率指标是一个重要创新，只有在用户编辑少于50%且不改变关键部分时，才视为接受。这一指标更准确地反映了用户对建议的真实接受程度。

关键设计：在设计中，Ansible Lightspeed采用了针对Ansible特定语法的优化模型，确保生成的代码符合YAML格式，并通过用户反馈不断迭代改进模型性能。

🖼️ 关键图片

fig_0
fig_1
fig_2

📊 实验亮点

实验结果显示，Ansible Lightspeed在多行Ansible任务建议中实现了49.08%的强接受率，且N-Day用户留存率在第30天达到13.66%。这些数据表明该工具在特定领域的有效性和用户满意度，显著优于现有通用代码助手。

🎯 应用场景

Ansible Lightspeed的潜在应用场景包括IT自动化、DevOps流程优化以及软件开发工具集成等。其能够显著提升开发者在使用Ansible时的效率，降低学习曲线，未来有望在更多领域推广，助力开发者更高效地完成任务。

📄 摘要（原文）

The availability of Large Language Models (LLMs) which can generate code, has made it possible to create tools that improve developer productivity. Integrated development environments or IDEs which developers use to write software are often used as an interface to interact with LLMs. Although many such tools have been released, almost all of them focus on general-purpose programming languages. Domain-specific languages, such as those crucial for Information Technology (IT) automation, have not received much attention. Ansible is one such YAML-based IT automation-specific language. Ansible Lightspeed is an LLM-based service designed explicitly to generate Ansible YAML, given natural language prompt. In this paper, we present the design and implementation of the Ansible Lightspeed service. We then evaluate its utility to developers using diverse indicators, including extended utilization, analysis of user edited suggestions, as well as user sentiments analysis. The evaluation is based on data collected for 10,696 real users including 3,910 returning users. The code for Ansible Lightspeed service and the analysis framework is made available for others to use. To our knowledge, our study is the first to involve thousands of users of code assistants for domain-specific languages. We are also the first code completion tool to present N-Day user retention figures, which is 13.66% on Day 30. We propose an improved version of user acceptance rate, called Strong Acceptance rate, where a suggestion is considered accepted only if less than 50% of it is edited and these edits do not change critical parts of the suggestion. By focusing on Ansible, Lightspeed is able to achieve a strong acceptance rate of 49.08% for multi-line Ansible task suggestions. With our findings we provide insights into the effectiveness of small, dedicated models in a domain-specific context.