Knowledge-aware Alert Aggregation in Large-scale Cloud Systems: a Hybrid Approach

📄 arXiv: 2403.06485v1 📥 PDF

作者: Jinxi Kuang, Jinyang Liu, Junjie Huang, Renyi Zhong, Jiazhen Gu, Lan Yu, Rui Tan, Zengyin Yang, Michael R. Lyu

分类: cs.SE, cs.CL, cs.LG

发布日期: 2024-03-11

备注: Accepted by Proceedings of the 46th International Conference on Software Engineering: Software Engineering in Practice (ICSE SEIP 2024)

DOI: 10.1145/3639477.3639745

💡 一句话要点

提出COLA以解决云系统中的警报聚合问题

🎯 匹配领域: 支柱九：具身大模型 (Embodied Foundation Models)

关键词: 云计算 警报聚合 故障检测 大型语言模型 相关性挖掘 混合方法 标准操作程序

📋 核心要点

  1. 现有警报聚合方法存在不足，无法有效处理大量相关警报，导致人工处理困难。
  2. 本文提出COLA，通过相关性挖掘和LLM推理相结合，利用外部知识提高警报聚合效率。
  3. COLA在三个大规模云平台数据集上测试，F1分数达到0.901至0.930，显著优于现有方法。

📝 摘要（中文）

由于云系统的规模和复杂性，系统故障会引发大量相关警报，形成“警报风暴”。虽然这些警报可以追溯到少数根本原因，但数量庞大使得人工处理变得不可行。因此，警报聚合对于帮助工程师集中精力于根本原因并促进故障解决至关重要。现有方法通常利用基于语义相似性或统计方法来聚合警报，但前者忽视了警报的因果关系，后者难以处理不频繁的警报。为了解决这些局限性，本文引入外部知识，即警报的标准操作程序(SOP)作为补充，提出了一种基于相关性挖掘和大型语言模型(LLM)推理的混合方法COLA。实验结果表明，COLA在处理大规模警报时表现出色，F1分数达到0.901至0.930，超越了现有最先进的方法。

🔬 方法详解

问题定义：本文旨在解决云系统中因系统故障引发的警报风暴问题。现有方法在处理大量相关警报时存在局限，无法有效聚合和分析这些警报。

核心思路：COLA的核心思路是结合相关性挖掘和大型语言模型推理，通过外部知识（如SOP）来增强警报聚合的准确性和效率。该设计旨在同时利用统计证据和推理能力，以应对复杂的警报场景。

技术框架：COLA的整体架构包括两个主要模块：相关性挖掘模块和LLM推理模块。相关性挖掘模块负责捕捉警报之间的时间和空间关系，并高效测量其相关性；而LLM推理模块则对低置信度的警报对进行详细分析。

关键创新：COLA的创新点在于其混合设计，既利用了统计方法处理频繁警报的优势，又结合了LLM的推理能力，确保在处理大规模警报时的整体效率。与传统方法相比，COLA能够更好地理解警报之间的因果关系。

关键设计：在COLA中，相关性挖掘模块采用高效的算法来捕捉警报的相关性，而LLM推理模块则通过特定的参数设置和损失函数来优化推理过程，确保对不确定警报对的深入分析。

🖼️ 关键图片

fig_0
fig_1
fig_2

📊 实验亮点

COLA在三个数据集上的实验结果显示，F1分数从0.901到0.930，显著优于现有最先进的方法，且在处理效率上表现出色。这表明COLA在实际应用中具有良好的性能和可行性。

🎯 应用场景

COLA的研究成果在云计算环境中具有广泛的应用潜力，能够有效提升故障检测和响应的效率，减少人工干预的需求。未来，该方法可扩展至其他复杂系统的警报管理和故障诊断领域，具有重要的实际价值。

📄 摘要（原文）

Due to the scale and complexity of cloud systems, a system failure would trigger an "alert storm", i.e., massive correlated alerts. Although these alerts can be traced back to a few root causes, the overwhelming number makes it infeasible for manual handling. Alert aggregation is thus critical to help engineers concentrate on the root cause and facilitate failure resolution. Existing methods typically utilize semantic similarity-based methods or statistical methods to aggregate alerts. However, semantic similarity-based methods overlook the causal rationale of alerts, while statistical methods can hardly handle infrequent alerts. To tackle these limitations, we introduce leveraging external knowledge, i.e., Standard Operation Procedure (SOP) of alerts as a supplement. We propose COLA, a novel hybrid approach based on correlation mining and LLM (Large Language Model) reasoning for online alert aggregation. The correlation mining module effectively captures the temporal and spatial relations between alerts, measuring their correlations in an efficient manner. Subsequently, only uncertain pairs with low confidence are forwarded to the LLM reasoning module for detailed analysis. This hybrid design harnesses both statistical evidence for frequent alerts and the reasoning capabilities of computationally intensive LLMs, ensuring the overall efficiency of COLA in handling large volumes of alerts in practical scenarios. We evaluate COLA on three datasets collected from the production environment of a large-scale cloud platform. The experimental results show COLA achieves F1-scores from 0.901 to 0.930, outperforming state-of-the-art methods and achieving comparable efficiency. We also share our experience in deploying COLA in our real-world cloud system, Cloud X.