Mergeable Model-Side Aggregation States for Long-Context Language Models

📄 arXiv: 2607.26448v1 📥 PDF

作者: Dachuan Song, Junyu Yin, Zechen Hu, Xuan Wang

分类: cs.CL, cs.AI

发布日期: 2026-07-29

🔗 代码/项目: GITHUB

💡 一句话要点

提出模型侧聚合状态以解决长上下文语言模型的聚合问题

🎯 匹配领域: 支柱九：具身大模型 (Embodied Foundation Models)

关键词: 长上下文语言模型 聚合任务 HyperLogLog 模型侧聚合 数据分析 智能助手

📋 核心要点

  1. 长上下文语言模型在处理集合聚合任务时，随着上下文长度的增加，性能显著下降。
  2. 本文提出了一种模型侧聚合接口，利用HLL草图状态来高效处理聚合任务，避免了额外的生成-执行-返回周期。
  3. 在多个实验中，该方法在准确率上显著优于传统的全上下文推理，提升幅度达到63.2个百分点。

📝 摘要（中文）

长上下文语言模型在处理非加法、基于集合的聚合任务时，随着上下文长度的增加，其性能变得不可靠。为此，本文提出了一种模型侧聚合接口，维护紧凑的基于哈希的HyperLogLog（HLL）草图状态。该方法通过提取器将每个相关记录映射到规范身份，并更新HLL状态。实验表明，该方法在多个聚合任务中表现出色，准确率接近100%。

🔬 方法详解

问题定义：本文旨在解决长上下文语言模型在处理非加法、集合聚合任务时性能不可靠的问题。现有方法在上下文长度增加时，聚合能力下降，导致结果不准确。

核心思路：提出一种模型侧聚合接口，维护紧凑的HLL草图状态，允许在处理上下文时高效更新聚合状态，避免了传统方法中的生成-执行-返回周期。

技术框架：整体架构包括一个冻结的语言模型和一个提取器，提取器将相关记录映射到规范身份，随后更新HLL状态。该状态可以在上下文段之间合并或直接读取用于下游推理。

关键创新：最重要的创新在于引入了HLL草图状态的合并能力，使得在多个上下文段中构建的状态可以无缝整合，显著提高了聚合任务的效率和准确性。

关键设计：HLL状态的大小设置为2 KiB（2048个寄存器），在上下文长度或集合基数增加时保持不变。实验中，HLL状态在处理一百万条记录时，平均相对误差仅为1.6%。

🖼️ 关键图片

fig_0
fig_1

📊 实验亮点

在实验中，该方法在3969个聚合后推理任务中达到了99.2%的准确率，相较于精确聚合的100.0%仅有0.8个百分点的差距。此外，在与传统全上下文推理的对比中，提升幅度达到63.2个百分点，显示出显著的性能优势。

🎯 应用场景

该研究的潜在应用领域包括日志分析、程序输出处理、表格数据聚合以及多轮对话系统等。通过提高长上下文语言模型在聚合任务上的性能，能够更好地支持智能助手、数据分析工具等应用，提升用户体验和决策效率。

📄 摘要（原文）

A known limitation of long-context language models is their increasingly unreliable performance in non-additive, set-based aggregation as context length grows. Examples include cardinality estimation, set relationships, and grouped statistics, which widely exist in logs, program outputs, tables, and multi-turn conversations. To provide the aggregation state required by these tasks, we introduce a model-side aggregation interface that maintains compact Hash-based HyperLogLog (HLL) sketch states alongside a frozen language model. While the model processes the context, an extractor maps each relevant record to a canonical identity. The identity is then hashed and updates the HLL state. These states can be merged across context segments and/or read out directly for downstream reasoning, avoiding an additional generate-execute-return cycle. We validate the proposed approach by setting the HLL state size as 2 KiB (2,048 registers), which does not increase with context length or set cardinality. In a distinct-count experiment involving one million records, the mean relative error was 1.6%. In a separate merge test, states built from as many as 256 segments produced exactly the same readout as a single pass over the same stream. On 3,969 aggregate-then-reason tasks from 174 source windows, the fixed-budget interface reached 99.2% accuracy on Gemma 4 (31B, BF16), compared with 100.0% under exact aggregation; the paired gap was 0.8 percentage points (95% window-cluster CI: 0.5-1.3 points). On a matched set of 174 items, our method improved over direct full-context reasoning by 63.2 points on Qwen and 56.3 points on Gemma. The corresponding gains over chain-of-thought (CoT) reasoning were 60.9 and 63.2 points, respectively. On a fixed 1,200-task Oolong-Synth subset, our method reached 91.1% on Qwen and 99.3% on Gemma. Code is available at https://github.com/songdc98/sketchops.