OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments

📄 arXiv: 2404.07972v2 📥 PDF

作者: Tianbao Xie, Danyang Zhang, Jixuan Chen, Xiaochuan Li, Siheng Zhao, Ruisheng Cao, Toh Jing Hua, Zhoujun Cheng, Dongchan Shin, Fangyu Lei, Yitao Liu, Yiheng Xu, Shuyan Zhou, Silvio Savarese, Caiming Xiong, Victor Zhong, Tao Yu

分类: cs.AI, cs.CL

发布日期: 2024-04-11 (更新: 2024-05-30)

备注: 51 pages, 21 figures

💡 一句话要点

提出OSWorld以解决多模态智能体在真实计算环境中的评估问题

🎯 匹配领域: 支柱九：具身大模型 (Embodied Foundation Models)

关键词: 多模态智能体 真实计算环境 任务评估 人机交互 开放式任务

📋 核心要点

  1. 现有基准缺乏互动环境，无法反映真实计算任务的多样性和复杂性，限制了智能体的评估和扩展性。
  2. OSWorld是一个可扩展的真实计算环境，支持多模态智能体的任务设置和执行评估，涵盖多个操作系统。
  3. 在OSWorld上进行的评估显示，现有的智能体在执行复杂任务时表现不佳，成功率仅为12.24%，远低于人类的72.36%。

📝 摘要（中文）

自主智能体能够在最小人类干预下完成复杂计算任务，具有变革人机交互的潜力。然而，现有基准缺乏互动环境或仅限于特定应用，无法反映真实计算使用的多样性与复杂性。为此，本文提出OSWorld，这是首个可扩展的真实计算环境，支持多模态智能体的任务设置、执行评估和互动学习。OSWorld包含369个真实的计算任务，涉及网页和桌面应用，提供可靠的评估机制。实验表明，现有的基于大语言模型的智能体在执行这些任务时表现不佳，揭示了其在GUI基础和操作知识上的不足。

🔬 方法详解

问题定义：本研究旨在解决现有多模态智能体评估基准缺乏真实互动环境的问题，现有方法无法有效反映真实计算任务的复杂性与多样性。

核心思路：提出OSWorld作为一个统一的计算环境，支持多模态智能体在真实应用场景中的任务设置和执行评估，旨在提高智能体的评估准确性和可扩展性。

技术框架：OSWorld的整体架构包括任务设置模块、执行评估模块和互动学习模块，支持在Ubuntu、Windows和macOS等多个操作系统上运行。

关键创新：OSWorld是首个可扩展的真实计算环境，能够支持任意应用程序的开放式计算任务评估，填补了现有基准的空白。

关键设计：每个任务示例都基于真实的计算用例，包含详细的初始状态配置和自定义的执行评估脚本，以确保评估的可靠性和可重复性。实验中使用的评估指标和数据集均公开可用。

🖼️ 关键图片

img_0

📊 实验亮点

在OSWorld上进行的评估显示，现有的基于大语言模型的智能体在执行369个复杂计算任务时，成功率仅为12.24%，而人类的成功率高达72.36%。这一结果揭示了智能体在图形用户界面理解和操作知识方面的显著不足。

🎯 应用场景

OSWorld的研究成果可广泛应用于智能助手、自动化办公、教育培训等领域，提升人机交互的效率和智能化水平。未来，OSWorld有望推动多模态智能体的进一步发展，促进更复杂任务的自动化处理。

📄 摘要（原文）

Autonomous agents that accomplish complex computer tasks with minimal human interventions have the potential to transform human-computer interaction, significantly enhancing accessibility and productivity. However, existing benchmarks either lack an interactive environment or are limited to environments specific to certain applications or domains, failing to reflect the diverse and complex nature of real-world computer use, thereby limiting the scope of tasks and agent scalability. To address this issue, we introduce OSWorld, the first-of-its-kind scalable, real computer environment for multimodal agents, supporting task setup, execution-based evaluation, and interactive learning across various operating systems such as Ubuntu, Windows, and macOS. OSWorld can serve as a unified, integrated computer environment for assessing open-ended computer tasks that involve arbitrary applications. Building upon OSWorld, we create a benchmark of 369 computer tasks involving real web and desktop apps in open domains, OS file I/O, and workflows spanning multiple applications. Each task example is derived from real-world computer use cases and includes a detailed initial state setup configuration and a custom execution-based evaluation script for reliable, reproducible evaluation. Extensive evaluation of state-of-the-art LLM/VLM-based agents on OSWorld reveals significant deficiencies in their ability to serve as computer assistants. While humans can accomplish over 72.36% of the tasks, the best model achieves only 12.24% success, primarily struggling with GUI grounding and operational knowledge. Comprehensive analysis using OSWorld provides valuable insights for developing multimodal generalist agents that were not possible with previous benchmarks. Our code, environment, baseline models, and data are publicly available at https://os-world.github.io.