EnvOS: Auditing Computer-Use Agents When the World Moves On : Phenomenon-Based Task Construction, Certification, and Evidence-Based Evaluation
作者: Ankush Kadu, Athira Gopal, Niharika Singh, Arpit Jain, Aswanth Krishnan
分类: cs.AI, cs.HC
发布日期: 2026-10-29
备注: 74 pages, 7 figures. Accepted at NeurIPS 2026 Workshops (AABA4ET and CL4FMAgents)
💡 一句话要点
提出EnvOS框架以解决动态环境下计算机代理的审计问题
🎯 匹配领域: 支柱二：RL算法与架构 (RL & Architecture) 支柱九：具身大模型 (Embodied Foundation Models)
关键词: 计算机代理 动态环境 环境审计 任务验证 强化学习 大型语言模型 智能系统
📋 核心要点
- 现有方法在动态环境中测试计算机代理时面临诸多挑战，代理的失败可能源于多种因素，难以定位问题。
- EnvOS框架通过在代理运行前后进行环境变化的验证和结果的重新审查，确保任务的可解性和评分的准确性。
- 实验结果显示，EnvOS在临床预约和保险索赔任务中显著提升了代理的成功率，尤其是在信息更新的情况下。
📝 摘要（中文）
计算机使用代理基于大型语言模型（LLMs）构建，面临在执行过程中环境变化的挑战。本文提出EnvOS框架，旨在构建和认证在任务执行中可变的环境。EnvOS在代理运行前检查环境变化的有效性，确保任务可解，并在运行后通过归档证据重新验证结果。实验表明，在临床预约和保险索赔场景中，EnvOS显著提高了代理的成功率，尤其是在信息更新时，展示了其在动态环境下的有效性和可靠性。
🔬 方法详解
问题定义：本文旨在解决计算机代理在动态环境中执行任务时的审计问题，现有方法难以准确判断代理失败的原因，可能是环境变化、信息更新或评分错误所致。
核心思路：EnvOS框架通过在代理执行前后对环境变化进行验证，确保每次变化的有效性和任务的可解性，从而提高代理的可靠性和准确性。
技术框架：EnvOS的整体架构包括环境变化的预检查模块、任务可解性验证模块和结果审查模块。每个模块在代理运行前后进行相应的检查和验证。
关键创新：EnvOS的主要创新在于其动态环境的审计能力，通过归档证据重新验证结果，确保评分的准确性，与传统静态环境测试方法有本质区别。
关键设计：EnvOS设计了自动化的验证器，确保每个环境都是从种子生成的，并通过审计的方式进行评分，适合用于强化学习中的可验证奖励训练。
🖼️ 关键图片
📊 实验亮点
在实验中，Claude Opus 5在未显示额外信息时成功率为0/16，提醒重新检查源后提升至2/16，而在显示更新值时成功率达到15/16，显著提升（p = 0.00293）。在保险索赔中，Opus 5成功率为10/10，Claude Sonnet 5从1/10提升至9/10，显示出EnvOS在动态环境下的有效性。
🎯 应用场景
EnvOS框架在多个领域具有广泛的应用潜力，尤其是在需要实时更新信息的场景，如医疗预约系统和保险索赔处理。通过提高代理在动态环境中的表现，EnvOS能够为智能代理的开发提供可靠的评估标准，推动智能系统的实际应用和发展。
📄 摘要（原文）
Computer-use agents built on large language models (LLMs) work in software that can change while they act: an appointment moves, a policy is revised, a record is updated. Testing agents under such changes is harder than it looks. When an agent fails, the cause may be the agent, but it may also be a change that did not fire as intended, an update the agent never received, or a grader that scored the episode wrongly. We introduce EnvOS, a framework for building and certifying environments that change mid-task. Before any agent runs, EnvOS checks that each change fires correctly, that the task remains solvable, and that the grader separates correct from incorrect behavior; after each run, separate code re-checks the verdict from archived evidence. We apply it to stale information, where a value the agent has already seen changes before the agent acts on it. In clinic booking, Claude Opus 5 succeeds on 0/16 tasks when nothing extra is shown, 2/16 when reminded to re-check the source, and 15/16 when the updated value is shown; the gain over the reminder holds after correcting for all tests (p = 0.00293). In insurance-claim approval, Opus 5 re-checks on its own and succeeds on 10/10, whereas Claude Sonnet 5 rises from 1/10 to 9/10 when the current policy is shown again, a gain significant in its pre-registered test but not after correcting for all tests. Whether an agent caught a change thus varied with the model and the interface. We also stress-test the grading itself: on corrupted episodes designed blind from the rules, the EnvOS pipeline accepted none of 26 as clean, and on an externally generated set it caught 25 of 39, against 4 for simple log checks. Because every EnvOS environment is generated from a seed and graded by an audited automatic verifier, the same environments are natural candidates for training agents with reinforcement learning from verifiable rewards.