ReAct Meets ActRe: When Language Agents Enjoy Training Data Autonomy

📄 arXiv: 2403.14589v3 📥 PDF

作者: Zonghan Yang, Peng Li, Ming Yan, Ji Zhang, Fei Huang, Yang Liu

分类: cs.AI, cs.CL, cs.LG

发布日期: 2024-03-21 (更新: 2024-04-01)

💡 一句话要点

提出A^3T框架以实现语言代理的自主训练数据生成

🎯 匹配领域: 支柱九：具身大模型 (Embodied Foundation Models)

关键词: 语言代理 自主学习 多步推理 行动轨迹 自我改进 策略梯度 对比自我训练

📋 核心要点

  1. 现有方法在收集多步推理和行动轨迹时仍需大量人工干预，限制了语言代理的自主性和效率。
  2. 本文提出A^3T框架，通过ActRe提示代理生成自主的代理轨迹，减少人工标注需求。
  3. 实验表明，A^3T代理在多个任务中显著超越现有技术，成功率接近人类水平，展示了强大的自我改进能力。

📝 摘要（中文）

语言代理通过与基础模型的推理展现了自主决策能力。近期的研究致力于通过多步推理和行动轨迹来提升语言代理的性能，但收集这些轨迹仍需大量人力。本文提出A^3T框架，利用ActRe提示代理生成自主的代理轨迹。通过随机采样外部动作，ReAct风格的代理可以查询ActRe代理以获取文本推理，从而合成新的轨迹。该方法通过策略梯度方法和二值化奖励实现对比自我训练，显著提升了代理的成功率。实验结果显示，使用A^3T的代理在AlfWorld中实现了96%的1-shot成功率，并在WebShop中表现接近人类专家。

🔬 方法详解

问题定义：本文旨在解决语言代理在多步推理和行动轨迹收集中的人工干预问题，现有方法依赖于人工标注或复杂的提示框架，效率低下。

核心思路：提出A^3T框架，利用ActRe提示代理生成自主的代理轨迹，通过随机采样和文本推理合成新轨迹，减少人工干预。

技术框架：A^3T框架包括两个主要模块：ReAct风格的代理和ActRe提示代理。ReAct代理负责执行动作并生成反馈，ActRe代理提供动作的文本推理。

关键创新：A^3T框架的创新在于通过ActRe代理生成的文本推理来合成新的行动轨迹，实现了语言代理的自主学习和自我改进，显著提高了成功率。

关键设计：采用策略梯度方法和二值化奖励进行对比自我训练，设计了多轮迭代的学习机制，使得代理能够在失败的任务中不断优化其策略。通过QLoRA微调，提升了模型的性能。

📊 实验亮点

实验结果显示，使用A^3T的代理在AlfWorld中实现了96%的1-shot成功率，并在经过4轮迭代后达到了100%的成功率。在WebShop中，A^3T代理的1-shot表现与人类平均水平相当，经过4轮迭代后接近人类专家的表现，显示出显著的性能提升。

🎯 应用场景

该研究的潜在应用领域包括智能助手、自动化客服和机器人控制等。通过减少人工干预，A^3T框架可以提高语言代理的自主性和效率，推动智能系统的进一步发展，具有广泛的实际价值和未来影响。

📄 摘要（原文）

Language agents have demonstrated autonomous decision-making abilities by reasoning with foundation models. Recently, efforts have been made to train language agents for performance improvement, with multi-step reasoning and action trajectories as the training data. However, collecting such trajectories still requires considerable human effort, by either artificial annotation or implementations of diverse prompting frameworks. In this work, we propose A$^3$T, a framework that enables the Autonomous Annotation of Agent Trajectories in the style of ReAct. The central role is an ActRe prompting agent, which explains the reason for an arbitrary action. When randomly sampling an external action, the ReAct-style agent could query the ActRe agent with the action to obtain its textual rationales. Novel trajectories are then synthesized by prepending the posterior reasoning from ActRe to the sampled action. In this way, the ReAct-style agent executes multiple trajectories for the failed tasks, and selects the successful ones to supplement its failed trajectory for contrastive self-training. Realized by policy gradient methods with binarized rewards, the contrastive self-training with accumulated trajectories facilitates a closed loop for multiple rounds of language agent self-improvement. We conduct experiments using QLoRA fine-tuning with the open-sourced Mistral-7B-Instruct-v0.2. In AlfWorld, the agent trained with A$^3$T obtains a 1-shot success rate of 96%, and 100% success with 4 iterative rounds. In WebShop, the 1-shot performance of the A$^3$T agent matches human average, and 4 rounds of iterative refinement lead to the performance approaching human experts. A$^3$T agents significantly outperform existing techniques, including prompting with GPT-4, advanced agent frameworks, and fully fine-tuned LLMs.