CAPRA: Scaling Feedback on Software Architecture Deliverables with a Multi-Agent LLM System

📄 arXiv: 2606.18976v1 📥 PDF

作者: Marco Becattini, Niccolò Caselli, Matteo Minin, Roberto Verdecchia, Enrico Vicario

分类: cs.SE, cs.AI

发布日期: 2026-06-17

备注: Accepted for publication at the 38th International Conference on Software Engineering Education and Training

💡 一句话要点

提出CAPRA以解决软件架构交付物反馈自动化问题

🎯 匹配领域: 支柱九：具身大模型 (Embodied Foundation Models)

关键词: 软件架构 自动化评估 多代理系统 大型语言模型 教育技术 反馈机制 文档提取

📋 核心要点

  1. 现有方法在软件架构交付物的评审中缺乏自动化，导致反馈效率低下和准确性不足。
  2. CAPRA通过多代理LLM系统，结合文本和UML图的解析，提供个性化的反馈，提升评审的自动化程度。
  3. 初步实验结果显示，CAPRA在88.8%的评估标准下表现良好，且处理速度快，显示出其在架构反馈中的潜力。

📝 摘要（中文）

在软件工程教育中，自动化评估已在代码评分和论文评分方面取得显著进展。然而，软件架构交付物的审查尚未完全自动化。CAPRA（可配置架构能力报告评估）是一个多代理LLM系统，旨在分析软件架构交付物，生成个性化的、符合模板的LaTeX反馈。CAPRA协调多个专业代理，利用Python微服务进行多模态文档提取，确保技术反馈的准确性和可靠性。初步实证评估显示，CAPRA在严格的评估标准下满足了88.8%的评估标准，处理每份报告的时间略超过4分钟。

🔬 方法详解

问题定义：论文旨在解决软件架构交付物的自动化评审问题，现有方法在结构完整性和需求追踪方面存在不足，导致反馈不够准确和可靠。

核心思路：CAPRA通过协调多个专业代理，利用大型语言模型（LLM）和多模态文档提取技术，生成符合模板的个性化反馈，从而提升评审的自动化和准确性。

技术框架：CAPRA的整体架构包括多个专门代理和一个基于Python的微服务，负责多模态文档提取，使用PyMuPDF和视觉增强的LLM（如gpt-4o）解析文本和UML图。

关键创新：CAPRA引入了确定性的证据锚定步骤，通过归一化的Levenshtein距离进行模糊匹配，确保反馈的可靠性，并通过ConsistencyManager代理进行交叉验证和结果合并，这是其与现有方法的本质区别。

关键设计：系统设计中采用了八个标准的二元评估分类法，涵盖提取完整性、特征验证、问题基础和严重性检测等方面，确保反馈的全面性和准确性。

🖼️ 关键图片

fig_0

📊 实验亮点

CAPRA在初步实验中显示出优异的性能，满足88.8%的评估标准，并与人类评估者的间接一致性达到中等水平（kappa = 0.582），每份报告处理时间仅略超过4分钟，展示了其在架构反馈中的有效性。

🎯 应用场景

CAPRA的研究成果可广泛应用于软件工程教育领域，尤其是在软件架构课程中，帮助教师和学生提高反馈效率和质量。未来，该系统还可扩展到其他领域的文档评审和自动化反馈，推动教育技术的发展。

📄 摘要（原文）

Automated assessment in software engineering education has advanced significantly for code grading and essay scoring. However, reviewing software architecture deliverables, which requires analyzing structural completeness and requirements traceability, has not yet been fully automated. Applying Large Language Models (LLMs) to this task requires robust architectures to ensure technical feedback is accurate and reliable for students. This paper presents CAPRA (Configurable Architecture Proficiency Report Assessment), a multi-agent LLM system that analyzes software architecture deliverables to generate personalized, template-compliant LaTeX feedback. As a core design choice, CAPRA coordinates multiple specialized agents and employs a Python-based microservice for multi-modal document extraction, utilizing PyMuPDF and vision-enabled LLMs (specifically gpt-4o) to parse text and UML diagrams. To ensure educational reliability and mitigate hallucinations, CAPRA introduces a deterministic Evidence Anchoring step using fuzzy matching via normalized Levenshtein distance, along with a ConsistencyManager agent that cross-verifies, deduplicates, and merges findings. System performance is assessed using a structured eight-criterion binary evaluation taxonomy covering: (i) extraction completeness, (ii) feature validation, (iii) issue grounding and severity detection, (iv) recommendation specificity and traceability, and (v) template and tone compliance. A preliminary empirical evaluation on 10 student reports shows that CAPRA satisfied 88.8% of the evaluated criteria under a strict two-rater aggregation rule, achieved moderate inter-rater agreement with human evaluators (kappa = 0.582), and processed each report in slightly over 4 minutes. While these results support the viability of LLM-supported architectural feedback, human oversight remains essential for subjective assessment dimensions.