Output Format Biases in the Evaluation of Large Language Models for Code Translation
作者: Marcos Macedo, Yuan Tian, Filipe R. Cogo, Bram Adams
分类: cs.SE, cs.AI
发布日期: 2024-03-25 (更新: 2025-10-13)
备注: This version (v2) is a journal extension of our previous conference submission that was accepted into the 2024 IEEE/ACM First International Conference on AI Foundation Models and Software Engineering (Forge 2024), which includes new experiments and results
💡 一句话要点
提出输出格式偏差的解决方案以改进代码翻译评估
🎯 匹配领域: 支柱九：具身大模型 (Embodied Foundation Models)
关键词: 代码翻译 大型语言模型 输出格式偏差 评估方法 提示工程 正则表达式 软件工程 性能评估
📋 核心要点
- 现有方法在评估LLMs的代码翻译性能时，未能有效处理输出中的非代码元素，导致评估结果偏差。
- 提出结合提示工程和正则表达式的策略，以提取混合格式输出中的源代码，从而减轻输出格式偏差。
- 实验证明，所提方法使得11个开源模型的代码提取成功率达到92.73%，显著提高了评估的可靠性。
📝 摘要（中文）
代码翻译在软件工程中至关重要，能够促进遗留系统的现代化、确保跨平台兼容性并提升软件性能。现有研究通过测试套件或参考输出评估大型语言模型（LLMs）的代码翻译性能，但生成的输出往往包含可执行代码和非代码元素，导致评估指标偏差。本文对11个开源LLMs在五种编程语言（C、C++、Go、Java和Python）上的输出进行了实证分析，发现26.4%至73.7%的输出需要后处理。为减轻输出格式偏差，提出了一种结合提示工程和正则表达式的策略，成功提取源代码，平均代码提取成功率达到92.73%。实验证实，输出格式偏差影响了计算准确性和文本基础指标的评估。
🔬 方法详解
问题定义：本文旨在解决在代码翻译任务中，现有评估方法因输出格式偏差而导致的性能评估不准确的问题。现有方法未能有效区分可执行代码与非代码元素，影响了评估的公正性。
核心思路：论文提出通过提示工程与正则表达式的结合，设计一种策略来提取源代码，减少非代码元素对评估结果的干扰。这种设计旨在提高代码提取的准确性和评估的可靠性。
技术框架：整体流程包括数据收集、输出格式分析、提示工程设计和正则表达式应用。首先对11个开源LLMs的输出进行分析，识别出非代码元素，然后设计相应的提示和正则表达式进行源代码提取。
关键创新：最重要的创新在于提出了一种新的评估方法，通过有效提取源代码，显著提高了代码翻译的评估准确性。这与传统方法的本质区别在于，传统方法未能考虑输出格式的多样性。
关键设计：在提示工程中，设计了特定的提示语以引导模型生成更清晰的代码输出；在正则表达式的设计上，针对不同编程语言的特性进行了优化，以提高代码提取的成功率。
🖼️ 关键图片
📊 实验亮点
实验结果显示，所提方法使得11个开源模型的代码提取成功率达到92.73%，显著高于传统方法。同时，输出格式偏差对计算准确性和文本基础指标（如BLEU、CodeBLEU和CrystalBLEU）的影响也得到了验证，强调了该研究的实际意义。
🎯 应用场景
该研究的潜在应用领域包括软件开发、代码迁移和自动化测试等。通过改进代码翻译的评估方法，能够更准确地评估和提升LLMs在实际应用中的表现，促进软件工程的效率和质量提升。未来，该方法可能对其他领域的模型评估产生积极影响。
📄 摘要（原文）
Code translation between programming languages (PLs) is a critical task in software engineering, facilitating the modernization of legacy systems, ensuring cross-platform compatibility, and enhancing software performance. Most existing studies instruct LLMs to perform code translation and evaluate their performance by either running the generated outputs through test suites or comparing them to reference outputs (ground truth). These outputs, however, may contain not only executable source code but also additional non-code elements, such as natural language explanations or formatting tokens. We refer to the combination of source code and non-code elements as the output format. It is crucial to understand and address variations in output format, as non-code elements can interfere with evaluation metrics, resulting in biased assessments of model performance and comparisons. We conduct an empirical analysis of the outputs from eleven instruct-tuned open-source LLMs, across five PLs: C, C++, Go, Java, and Python. The results show that between 26.4% and 73.7% of outputs produced by our evaluated LLMs necessitate post-processing. To mitigate output format bias, we propose a strategic combination of prompt engineering and regular expressions that effectively extracts source code from mixed-format outputs, enabling the eleven open-source models to achieve an average Code Extraction Success Rate (CSR) of 92.73%. Our empirical study confirms that output format bias affects widely used execution-based metrics, i.e., Computational Accuracy (CA), and text-based metrics, i.e., BLEU, CodeBLEU and CrystalBLEU. Additionally, we test five closed-source LLMs and observe that they also generate varying distributions of output formats, which could lead to output format biases. Our results highlight the need to mitigate the output format bias to enable reliable evaluations in LLMs for code translation.