From Routes to Steps: Separating Semantic Progress from Local Execution in Vision-and-Language Navigation

📄 arXiv: 2608.03143v1 📥 PDF

作者: Xiangyun Huang, Xiangchen Wang, Runfeng Lin, Yihao Xu, Kangyu Huang, Jiang Hengchen, Xiwang Dong, Lin Jiarong

分类: cs.CV, cs.RO

发布日期: 2026-08-04

备注: 16 pages, 9 figures

🔗 代码/项目: PROJECT_PAGE

💡 一句话要点

提出Route2Step以解决视觉语言导航中的进度跟踪与执行错误问题

🎯 匹配领域: 支柱六：视频提取与匹配 (Video Extraction) 支柱九：具身大模型 (Embodied Foundation Models)

关键词: 视觉语言导航 进度跟踪 动作生成 多模态学习 智能体决策

📋 核心要点

  1. 现有的视觉语言导航方法难以区分进度跟踪错误与执行错误，导致智能体在错误状态下继续决策。
  2. 本文提出Route2Step框架，通过步骤级接口将语义进度跟踪与动作生成解耦，提升导航准确性。
  3. 在R2R-CE数据集上，Route2Step的成功率从48.1%提升至55.3%，路径长度比从43.3%提升至48.2%。

📝 摘要（中文）

视觉语言导航（VLN）要求智能体根据路线级指令，通过执行其组成步骤来完成导航。现有的基于视觉语言模型的导航器通常仅通过下一步预测来监督这两种能力，导致进度跟踪错误与执行错误难以区分。为了解决这一模糊性，本文提出了Route2Step框架，通过显式的步骤级接口将语义进度跟踪与动作生成解耦。该框架包括指令分析模块和动作生成模块，利用步骤对齐程序来监督进度状态。实验结果表明，Route2Step在R2R-CE数据集上显著提升了成功率和路径长度比，展示了其在实际环境中的应用潜力。

🔬 方法详解

问题定义：本文旨在解决视觉语言导航中进度跟踪与执行错误难以区分的问题。现有方法仅依赖下一步预测，导致智能体在错误的进度状态下做出决策，影响导航效果。

核心思路：Route2Step框架通过引入步骤级接口，将语义进度跟踪与动作生成解耦。通过分析全局指令和视觉历史，明确区分进度状态与动作生成，从而提高导航的准确性和鲁棒性。

技术框架：该框架主要包括两个模块：指令分析模块（$ ext{M}{ ext{IA}}$）用于预测进度状态，动作生成模块（$ ext{M}{ ext{AG}}$）根据预测状态和最近观察生成局部动作。步骤对齐程序（E-SPA）用于监督进度状态。

关键创新：最重要的创新在于通过步骤级接口解耦进度跟踪与动作生成，使得智能体能够更准确地判断当前状态与执行动作的关系。这一设计显著提高了导航的成功率。

关键设计：在训练过程中，使用190K状态级纠正样本进行进度监督，仅需11.5K直接动作监督状态。步骤对齐程序有效地将子指令与路线级演示对应，确保了进度状态的准确监督。

🖼️ 关键图片

fig_0
fig_1
fig_2

📊 实验亮点

Route2Step在R2R-CE数据集上的实验结果显示，成功率从48.1%提升至55.3%，路径长度比从43.3%提升至48.2%。这些结果表明，该方法在实际导航任务中具有显著的性能提升，验证了其有效性。

🎯 应用场景

该研究的潜在应用领域包括智能家居、机器人导航、增强现实等场景，能够提升智能体在复杂环境中的自主导航能力。未来，Route2Step框架有望在更广泛的视觉语言理解任务中发挥作用，推动多模态交互技术的发展。

📄 摘要（原文）

Vision-and-Language Navigation (VLN) requires an agent to follow a route-level instruction by executing its constituent steps from egocentric visual observations. Existing VLM-based navigators typically supervise both capabilities through next-action prediction alone, making progress-tracking errors difficult to distinguish from execution errors. When an agent deviates from the route, a corrective action label may recover the next movement but does not indicate whether the agent selected the wrong sub-instruction or failed to execute the correct one. Consequently, the agent may continue making decisions from an erroneous progress state. To resolve this ambiguity, we propose \textbf{Route2Step}, a framework that decouples semantic progress tracking from action generation through an explicit step-level interface. The Instruction Analysis Module ($\mathcal{M}{\mathrm{IA}}$) predicts this state from the global instruction and visual history. Conditioned on the predicted state and recent observations, the Action Generation Module ($\mathcal{M}{\mathrm{AG}}$) generates local action chunks. To supervise the progress state without manual temporal labels, E-SPA, a step-alignment procedure, associates sub-instructions with their corresponding portions of route-level demonstrations. These alignments enable state supervision for incorrect progress estimates, while direct action supervision is reserved for rollout groups that repeatedly fail under the correct active sub-instruction. On R2R-CE, Route2Step improves SR from 48.1\% to 55.3\% and SPL from 43.3\% to 48.2\%, using 190K state-level corrective samples while requiring only 11.5K directly action-supervised states. Experiments in real-world indoor and outdoor environments further demonstrate the practical applicability of Route2Step. The project page is: https://sisyphus-hxy.github.io/Route2Step/.