What Does "Now" Mean to Your Computer-Use Agent? Stale Screenshots, Page Loads, and Which Check to Run Before a Click
作者: Vasundra Srinivasan
分类: cs.AI
发布日期: 2026-10-29
备注: 8 pages, 4 figures, 3 tables. Code, benchmark generator and preregistrations: https://github.com/multimodal-ai-engineering/preflight-bench
🔗 代码/项目: GITHUB
💡 一句话要点
提出基于像素比较的检查机制以解决点击失效问题
🎯 匹配领域: 支柱九：具身大模型 (Embodied Foundation Models)
关键词: 点击验证 页面加载 像素比较 计算机代理 用户体验 自动化测试 动态网页
📋 核心要点
- 现有方法未能有效处理页面加载导致的点击失效问题，尤其是在页面刚打开时。
- 论文提出通过像素比较来验证点击目标位置，显著降低点击失效的风险。
- 实验表明，等待网络活动停止可以将点击失效率降低至1%以下，且新方法在验证成功率上表现优异。
📝 摘要（中文）
计算机使用代理在屏幕截图上规划点击，但如果页面在此期间发生变化，点击可能会落在意外元素上。研究发现，页面加载是导致点击失效的主要风险，尤其是在页面刚打开后的前几秒。通过等待网络活动停止，可以显著降低这一风险。论文提出了一种像素比较的方法，能够在低延迟下验证目标位置，并与现有的前沿视觉语言模型相媲美。研究还发布了基准生成器和代理开发者的三条规则。
🔬 方法详解
问题定义：论文要解决的问题是计算机使用代理在页面加载过程中点击失效的风险，现有方法未能有效应对页面动态变化带来的挑战。
核心思路：论文的核心解决思路是通过在点击前进行像素比较，验证目标元素的位置，从而避免点击失效。这样的设计能够在较低的延迟下提供准确的验证。
技术框架：整体架构包括页面加载监测、网络活动检测和点击前验证三个主要模块。首先监测页面加载状态，然后在网络活动停止后进行目标验证。
关键创新：最重要的技术创新点是引入像素比较方法，能够在一秒内准确验证目标位置，且延迟仅为现有视觉语言模型的八分之一。
关键设计：关键设计包括网络活动监测的时间阈值设置，以及像素比较算法的实现细节，确保在动态页面中也能快速响应并验证点击目标。
🖼️ 关键图片
📊 实验亮点
实验结果显示，在31个公共网页上，等待网络活动停止后，点击失效率从17%降至0.4%。此外，像素比较方法在验证目标位置时的延迟仅为现有模型的八分之一，显示出显著的性能提升。
🎯 应用场景
该研究的潜在应用领域包括自动化测试、网页交互代理和智能助手等。通过减少点击失效的风险，可以提高用户体验和操作效率，特别是在动态网页环境中。未来，该方法可能会被广泛应用于各种需要实时交互的计算机应用中。
📄 摘要（原文）
Computer-use agents plan on a screenshot for several seconds and then click at a position on the screen; if the page has changed in the meantime, the click lands on an unintended element. Prior work treats the planning interval as the source of this risk. Measuring planned clicks on 31 public web pages, we find that the risk lies in page loading: 0.4% of clicks planned on fully loaded pages became stale within 30 seconds, against 17% of clicks planned in the first second after a page opened. On these pages, waiting about one second for network activity to stop removes most of this risk, yet the wait rules of eight of ten widely used open-source agents leave more than 1% of clicks stale. Commercial pages keep loading for longer: in the HTTP Archive's public crawl, 15% of 65 desktop booking and buying home pages went 500 ms without network requests within 3 seconds of their HTML being parsed, against 63% of 3,028 .gov pages, and 52% shifted their layout within 6.5 seconds of a network-idle wait capped at 3 seconds, against 8% (61% against 26% among pages recorded in full). Where a check before the click is needed, as before irreversible actions and on pages that are still loading, a pixel comparison verifies the target's position as accurately as a frontier vision-language model at one eighth of the latency, and a 4B model matches the frontier model in verifying whether the previous action succeeded. We release a benchmark generator, a break-even formula for when a check pays, and three rules for agent developers: https://github.com/multimodal-ai-engineering/preflight-bench