BiliVLA: Scene-Aware Vision-Language-Action Model with Reinforcement Learning for Autonomous Biliary Endoscopic Navigation

📄 arXiv: 2606.23531v1 📥 PDF

作者: Jinsong Lin, Chi kit Ng, Zhiyong Xiong, Zikang Pan, Yihan Hu, Tabassum Tamima, Ziyi Hao, Eddie Cheung, Jiewen Lai, Huxin Gao, Hongliang Ren

分类: cs.RO

发布日期: 2026-06-22

💡 一句话要点

提出BiliVLA以解决自主胆道内窥镜导航问题

🎯 匹配领域: 支柱二：RL算法与架构 (RL & Architecture) 支柱九：具身大模型 (Embodied Foundation Models)

关键词: 内窥镜导航 视觉-语言-动作 深度学习 强化学习 医疗机器人 场景感知 语义一致性

📋 核心要点

  1. 现有的内窥镜导航系统在解剖变异和视觉伪影下表现不佳，限制了其在胆道插管中的自主性和可靠性。
  2. BiliVLA框架通过将内窥镜导航视为基于指令的视觉运动学习问题，结合场景感知和奖励引导优化，提升了导航精度。
  3. 在三项ERCP子任务中，BiliVLA实现了91.96%的平均动作精度和84.85%的成功率，显著提高了操作的可靠性。

📝 摘要（中文）

胆道逆行胰胆管造影（ERCP）要求在狭窄的单目视野中进行精确的内窥镜导航和稳定的胆道插管。尽管近期的机器人系统和基于视觉的辅助技术改善了操作员的人体工程学并提供了感知线索，但在显著的解剖变异和安全关键的视觉伪影下，其性能下降，妨碍了插管级程序的可靠自主性。本文提出了BiliVLA，一个场景感知的视觉-语言-动作（VLA）框架，将胆道内窥镜导航表述为一个基于指令的视觉运动学习问题。BiliVLA在给定内窥镜观察和阶段特定语言指令的情况下，联合预测目标类别、定位的边界框和连续内窥镜的离散三自由度（DoF）运动指令。该框架通过场景感知监督增强语义目标一致性，并通过安全感知恢复监督诱导在腔壁接触下的保守撤退行为。实验结果表明，BiliVLA在真实的幻影实验中实现了91.96%的平均动作精度和84.85%的整体成功率。

🔬 方法详解

问题定义：本文旨在解决胆道内窥镜导航中的自主性和可靠性问题，现有方法在面对解剖变异和视觉伪影时表现不佳，影响了插管的成功率。

核心思路：BiliVLA框架通过将内窥镜导航视为一个指令条件的视觉运动学习问题，利用场景感知和奖励引导优化来提升导航的精确性和安全性。

技术框架：BiliVLA的整体架构包括三个主要模块：输入的内窥镜观察、阶段特定的语言指令处理和联合预测目标类别及运动指令的决策模块。

关键创新：BiliVLA的创新点在于引入了场景感知监督和安全感知恢复监督，显著提升了语义目标一致性和在腔壁接触情况下的保守行为。

关键设计：该框架采用了两阶段的训练范式，结合了增强的监督微调（SFT）和群体相对策略优化（GRPO），以提高动作的可靠性和决策的一致性。

🖼️ 关键图片

fig_0
fig_1
fig_2

📊 实验亮点

BiliVLA在三项ERCP子任务中实现了91.96%的平均动作精度和84.85%的整体成功率，较现有方法有显著提升，表明其在自主内窥镜导航中的有效性和可靠性。

🎯 应用场景

BiliVLA的研究成果在医疗内窥镜导航领域具有重要的应用潜力，能够提升胆道插管的成功率和安全性，减少操作风险。未来，该技术有望推广至其他类型的内窥镜手术，进一步改善手术效果和患者安全。

📄 摘要（原文）

Endoscopic retrograde cholangiopancreatography (ERCP) demands precise endoscopic navigation and stable biliary cannulation within a narrow monocular field characterized by specular reflections, partial occlusions, and frequent tissue contact. Although recent robotic systems and vision-based assistance techniques improve operator ergonomics and provide perceptual cues, their performance degrades under pronounced anatomical variability and safety-critical visual artifacts, which hinders reliable autonomy in cannulation-grade procedures. Here, we present BiliVLA, a scene-aware Vision-Language-Action (VLA) framework that formulates biliary endoscopic navigation as an instruction-conditioned visuomotor learning problem. Given an endoscopic observation and a stage-specific language instruction, BiliVLA jointly predicts the target category, a grounded bounding box, and a discrete three degrees of freedom (DoF) motor command for a continuum endoscope. The proposed framework incorporates scene-aware supervision to enhance semantic target consistency and safety-aware recovery supervision to induce conservative retreat behaviors under luminal wall contact. A key component of BiliVLA is a two-stage training paradigm that combines grounding-enhanced supervised fine-tuning (SFT) with Group Relative Policy Optimization (GRPO), which significantly improves action reliability and decision consistency during closed-loop navigation. Across three ERCP subtasks, BiliVLA achieves an average action precision of 91.96\% and an overall success rate (SR) of 84.85\% in real-world phantom experiments. These results indicate that integrating semantic grounding, scene-aware learning, and reward-guided optimization improves perception-action alignment and enables robust autonomous endoscopic navigation.