From Static to Dynamic: Benchmarking Real-World Code Review with MCR-Bench

📄 arXiv: 2608.27442v1 📥 PDF

作者: Dewu Zheng, Yanlin Wang, Xiwen Wang, Kefeng Duan, Hongyu Zhang, Xilin Liu, Yuchi Ma, Zibin Zheng

分类: cs.SE, cs.AI, cs.CL

发布日期: 2026-08-27

备注: Accepted at ISSTA 2026

💡 一句话要点

提出MCR-Bench以解决多轮代码审查的真实场景问题

🎯 匹配领域: 支柱九：具身大模型 (Embodied Foundation Models)

关键词: 代码审查 缺陷检测 多轮互动 大型语言模型 软件质量保证 基准测试 动态状态感知

📋 核心要点

  1. 现有方法通常将代码审查简化为单轮决策，无法反映真实开发中的多轮互动和复杂性。
  2. 本文提出MCR-Bench基准，专注于多轮代码审查，提供缺陷状态感知和细粒度的缺陷元数据。
  3. 实验结果显示主流LLMs在缺陷检测和生命周期状态跟踪方面表现有限，且随着互动轮次增加性能显著下降。

📝 摘要（中文）

在真实的软件开发中，代码审查通常涉及开发者与审查者之间的多轮互动，以提高软件质量，这一过程既耗时又昂贵。尽管近期有研究探索大型语言模型（LLMs）用于自动化代码审查，但大多数方法将代码审查简化为单轮静态决策任务，未能捕捉到真实审查场景中的多轮互动特性和复杂问题解决过程。为此，本文提出了MCR-Bench，这是第一个针对真实多轮代码审查设计的缺陷状态感知基准，涵盖五种常用编程语言，包含2269个真实的多轮代码审查任务，并附有细粒度的缺陷信息和跨轮状态标签。

🔬 方法详解

问题定义：本文旨在解决现有代码审查方法过于简化的问题，特别是缺乏对多轮互动和复杂缺陷处理的关注。现有方法未能有效捕捉真实场景中的动态变化和交互过程。

核心思路：MCR-Bench通过提供真实的多轮代码审查任务和细粒度的缺陷信息，旨在更好地模拟真实开发环境中的代码审查过程，从而提升LLMs在此任务上的表现。

技术框架：MCR-Bench的整体架构包括任务生成模块、缺陷注释模块和状态跟踪模块。每个任务都包含缺陷描述、类型和严重性等元数据，并标注跨轮状态，以便于分析和评估。

关键创新：MCR-Bench的主要创新在于其缺陷状态感知的设计，使其能够捕捉缺陷在多轮审查过程中的演变轨迹，这与现有方法的静态决策模式形成鲜明对比。

关键设计：在设计中，MCR-Bench采用了细粒度的缺陷元数据和动态状态注释，确保每个任务能够反映缺陷的完整生命周期。此外，实验中使用了多种主流LLMs进行性能评估，揭示了不同缺陷类型和严重性对模型表现的影响。

🖼️ 关键图片

fig_0
fig_1
fig_2

📊 实验亮点

实验结果表明，主流LLMs在缺陷检测和生命周期状态跟踪方面的整体能力有限，随着互动轮次的增加，性能显著下降。特别是在处理语义复杂或低显著性的缺陷时，模型的漏检率显著提高，显示出跨轮时间对齐和长程记忆不足等关键弱点。

🎯 应用场景

MCR-Bench的研究成果可广泛应用于软件开发领域，尤其是在代码审查和质量保证过程中。通过提升自动化代码审查工具的性能，能够有效降低开发成本，提高软件质量。此外，该基准的设计也为后续研究提供了重要的参考框架，推动了代码审查领域的进一步发展。

📄 摘要（原文）

In real-world software development, code review typically involves iterative interactions between developers and reviewers to improve software quality, making the process costly and time-consuming. Although recent work explores large language models (LLMs) for automated code review, most approaches oversimplify code review into a single-round, static decision task, which fails to capture the multi-round interactive nature and the complex problem-solving processes inherent in realistic review scenarios. To bridge this gap, we introduce MCR-Bench, the first defect state-aware benchmark designed for realistic multi-round code review. MCR-Bench covers five commonly-used programming languages and consists of 2,269 real-world multi-round code review tasks, each of which is annotated with fine-grained defect information and cross-round state labels. Each task in MCR-Bench is equipped with fine-grained defect metadata (e.g., description, type, severity) alongside dynamic state annotations, capturing the complete evolutionary trajectory of a defect throughout the multi-round process. We obtain several findings through extensive experiments on MCR-Bench with mainstream LLMs. (1) Limited overall capability: experiments reveal that mainstream LLMs exhibit limited overall performance in defect detection and defect lifecycle state tracking, with performance degrading significantly as the number of interaction rounds increases; (2) Defect-sensitive performance: LLMs' performance varies substantially across different defect types and severity levels, with semantically complex or low-salience defects being significantly more likely to be missed; (3) Underlying Failure Mechanisms: our in-depth error analysis dissects the distinct drivers of false positives and false negatives, revealing critical weaknesses such as cross-round temporal misalignment and inadequate long-range memory.