Adversarial Speculative Reasoning Improves Long Horizon Agents

📄 arXiv: 2611.05345v1 📥 PDF

作者: Hongyin Luo, Wei Fang, Lijie Yang, Owen Stepan, Daniel Suit, Jeff Krapf, Dana Wensberg, Jack O'Brien

分类: cs.AI

发布日期: 2026-11-03

💡 一句话要点

提出对抗性推测推理以提升长时间跨度智能体的推理能力

🎯 匹配领域: 支柱九：具身大模型 (Embodied Foundation Models)

关键词: 长时间跨度推理 推测解码 对抗性训练 智能体模型 噪声控制 自动化决策 复杂任务

📋 核心要点

  1. 现有的推测解码方法未能有效解决长时间跨度智能体推理中的数值噪声问题，导致推理准确性下降。
  2. 本文提出通过对抗性偏好训练草稿模型，以控制推理过程中的噪声，从而提升目标模型的推理能力。
  3. 在复杂的智能体编码基准上，所提方法提高了端到端成功率，提升幅度在0.06%到0.28%之间。

📝 摘要（中文）

推测解码通常被视为一种系统优化手段，旨在加速大型语言模型的解码过程，而不改变模型的行为。然而，针对长时间跨度智能体推理的研究表明，推测解码的效果并非如此简单。本文提出通过控制数值噪声来提升目标模型的推理能力，尽管无法改变目标模型的整体标记分布，但可以通过训练草稿模型以拒绝不必要的推理，从而在关键决策点进行更精确的推理。实验结果表明，该方法在复杂的智能体编码基准上提高了端到端成功率，而无需改变目标模型或其输出分布。

🔬 方法详解

问题定义：本文旨在解决长时间跨度智能体推理中，由于推测解码导致的数值噪声问题，现有方法未能有效控制噪声，影响推理准确性。

核心思路：通过训练草稿模型以拒绝不必要的推理，控制推理过程中的噪声，从而在关键决策点进行更精确的推理，提升目标模型的推理能力。

技术框架：整体架构包括草稿模型和目标模型两个部分。在推理过程中，草稿模型在关键决策点与目标模型分开，以确保在最小噪声条件下进行推理。

关键创新：最重要的创新点在于将推测解码机制从单纯的加速器转变为主动分配器，提升上下文精度，从而改善长时间跨度智能体的推理准确性。

关键设计：在训练过程中，草稿模型的参数设置和损失函数设计旨在最大限度地减少噪声影响，确保在关键决策点进行高精度的推理。

🖼️ 关键图片

img_0
img_1

📊 实验亮点

实验结果显示，所提方法在复杂智能体编码基准上显著提高了端到端成功率，提升幅度在0.06%到0.28%之间，且未对目标模型或其输出分布进行任何修改，展现了方法的有效性。

🎯 应用场景

该研究的潜在应用领域包括复杂任务的自动化决策系统、智能体控制和长时间跨度的推理任务。通过提升智能体的推理能力，能够在更复杂的环境中实现更高效的决策，具有重要的实际价值和未来影响。

📄 摘要（原文）

Speculative decoding has been treated purely as a systems optimization that speeds up large language model decoding without changing the model behavior. Under this view, the growing effort to train better draft models has been focused solely on acceptance rate. We argue that this view does not hold true for long-horizon agent reasoning. Measuring frontier open-weight models on complex agentic tasks with different speculative decoding approaches, we found that numerical noise grows within verification windows, and the draft acceptance results of the thinking process can change the output distribution of the model after 30 steps, thus noticeably influencing the reasoning accuracy of long-horizon agents in random directions. In this work, we improve the reasoning capability of target models by controlling such noise. While we cannot change the overall token distribution verified by the target model, we can encode critical reasoning with highest precision by training the draft model to be rejected. We identify these points in agent traces and train the draft model to favor adversarial preference. At inference the trained draft and the target part ways precisely at these decision points, verification restarts, so that critical thinking tokens are sampled from the target model and encoded under its least noisy conditions. This approach makes critical reasoning better indexed and more durable over a long horizon. On complex agentic coding benchmarks, our approach raises end-to-end success rate with no change to the target model or its output distribution. Our results recast the speculative decoding mechanism from a pure accelerator into an active allocator for context precision that improves the accuracy of long-horizon agents by training 0.06% to 0.28% of hosted parameters.