Selective Agent Guidance via Entropy: Learning Autonomous Policies from Imperfect VLM Teachers

📄 arXiv: 2609.01567v1 📥 PDF

作者: Matteo Merler, Giovanni Bonetta, Davide Zago, Rossella Cancelliere, Bernardo Magnini

分类: cs.AI, cs.CL, cs.LG

发布日期: 2026-09-01

备注: 9 pages, 3 figures, 4 tables in the main text, 27 pages, 4 figures, 9 tables including Appendix

期刊: EMNLP 2026 Findings

💡 一句话要点

提出SAGE框架以从不完美的VLM教师学习自主策略

🎯 匹配领域: 支柱二：RL算法与架构 (RL & Architecture)

关键词: 视觉-语言模型 自主学习 强化学习 策略蒸馏 不确定性评估 多模态学习 决策系统

📋 核心要点

  1. 现有的视觉-语言模型（VLMs）在决策过程中使用时成本高且容易出错，无法从环境交互中学习。
  2. 本文提出SAGE框架，通过在学习者不确定时查询VLM，利用环境优势加权教师动作蒸馏，从而学习廉价的自主策略。
  3. 实验结果显示，SAGE在多个环境中表现优于无指导的强化学习，尤其在高奖励轨迹的发现上具有显著优势。

📝 摘要（中文）

视觉-语言模型（VLMs）为交互式决策提供了有用的先验知识，但直接将其作为策略使用成本高且不稳定。本文研究如何从在线、昂贵且不完美但信息丰富的VLM教师中学习廉价的自主策略。我们提出了SAGE（Selective Agent Guidance via Entropy）框架，该框架仅在学习者不确定时查询VLM，在训练期间执行建议的动作，并将指导蒸馏为轻量级的强化学习（RL）策略。实验结果表明，SAGE在多个环境中学习的策略在评估时无需VLM指导，并且在稀疏奖励的视觉推理和导航任务中优于无指导的RL。

🔬 方法详解

问题定义：本文解决的问题是如何从不完美的视觉-语言模型（VLM）教师中学习有效的自主策略。现有方法在每一步都需查询VLM，导致高成本和系统性错误。

核心思路：SAGE框架的核心思想是仅在学习者不确定时查询VLM，并在训练中执行建议的动作，同时将这些指导信息蒸馏为轻量级的强化学习策略。这样设计的目的是降低VLM的使用频率，提高学习效率。

技术框架：SAGE的整体架构包括三个主要模块：不确定性评估模块、VLM查询模块和策略蒸馏模块。学习者在每个训练步骤中评估自身的不确定性，只有在不确定时才会向VLM查询建议。

关键创新：SAGE的主要创新在于通过环境反馈加权教师动作蒸馏，而不是将所有建议视为同等有用。这一方法使得学习者能够更有效地利用VLM的指导。

关键设计：在SAGE中，关键的参数设置包括不确定性阈值的选择和教师动作的加权策略。此外，损失函数设计上考虑了环境反馈，以优化策略的学习过程。整体网络结构采用了轻量级的强化学习框架，以适应实时决策需求。

📊 实验亮点

实验结果表明，SAGE在多个稀疏奖励的视觉推理和导航任务中，学习的策略在评估时无需VLM指导，且在某些环境中超越了其VLM教师，显示出显著的性能提升。

🎯 应用场景

该研究的潜在应用领域包括机器人导航、智能助手和自动化决策系统等。通过有效利用视觉-语言模型的指导，SAGE能够在复杂环境中实现更高效的自主学习，具有重要的实际价值和广泛的应用前景。

📄 摘要（原文）

Vision-Language Models (VLMs) provide useful priors for interactive decision-making, but using them directly as policies is expensive and brittle: they must be queried at every step, do not improve from environment interaction, and can repeat systematic errors. We study how to learn a cheap autonomous policy from an online, expensive, and imperfect but informative VLM teacher. We propose SAGE (Selective Agent Guidance via Entropy), a framework that queries a VLM only when the learner is uncertain, executes the suggested action during training, and distills guidance into a lightweight Reinforcement Learning (RL) policy. Because VLM advice is not always reliable, SAGE can weight teacher-action distillation using environment-derived advantages rather than treating all suggestions as equally useful. Across sparse-reward visual reasoning and navigation tasks, SAGE learns policies that act without VLM guidance at evaluation time and improves over unguided RL in several environments, including settings where the learned policy exceeds its VLM teacher. The results show that selective guidance is most beneficial when the VLM can help the agent discover high-reward trajectories, and less useful when unguided exploration already succeeds or teacher actions do not lead to informative experience. SAGE also reduces VLM usage by prompting the teacher only on a fraction of training steps and requiring no VLM calls at deployment. Overall, our results suggest that VLMs don't need to be used as fixed policies to be useful; they can instead act as temporary, imperfect sources of guidance whose value is tested and internalized through interaction.