Surgical-DINO: Adapter Learning of Foundation Models for Depth Estimation in Endoscopic Surgery

📄 arXiv: 2401.06013v2 📥 PDF

作者: Beilei Cui, Mobarakol Islam, Long Bai, Hongliang Ren

分类: cs.CV, cs.AI

发布日期: 2024-01-11 (更新: 2024-01-12)

备注: Accepted by IPCAI 2024 (IJCAR Special Issue)

🔗 代码/项目: GITHUB

💡 一句话要点

提出Surgical-DINO以解决内窥镜手术中的深度估计问题

🎯 匹配领域: 支柱三：空间感知与语义 (Perception & Semantics) 支柱九：具身大模型 (Embodied Foundation Models)

关键词: 深度估计 内窥镜手术 基础模型 低秩适应 DINOv2 医疗影像 机器学习 计算机视觉

📋 核心要点

  1. 现有的基础模型在医疗和手术领域的深度估计应用中存在性能不足的问题，无法直接满足特定需求。
  2. 本文提出了一种名为Surgical-DINO的低秩适应方法，通过集成LoRA层来适应手术特定领域知识，避免传统的微调方式。
  3. 在MICCAI挑战数据集SCARED上进行的实验表明，Surgical-DINO在内窥镜深度估计任务中显著优于所有现有模型。

📝 摘要（中文）

本研究旨在解决内窥镜手术中的深度估计问题，这对3D重建、手术导航和增强现实可视化至关重要。尽管基础模型在许多视觉任务中表现出色，但在医疗和手术领域的应用中存在局限性。本文提出了一种低秩适应（LoRA）的方法，将DINOv2模型应用于手术深度估计。通过冻结DINO图像编码器，仅优化LoRA层和深度解码器，实验结果表明Surgical-DINO在内窥镜深度估计任务中显著优于现有的最先进模型。

🔬 方法详解

问题定义：本研究旨在解决内窥镜手术中的深度估计问题。现有的基础模型在医疗领域的应用中表现不佳，尤其是在特定的手术场景下，无法有效提取深度信息。

核心思路：本文提出的Surgical-DINO通过低秩适应（LoRA）方法，结合DINOv2模型，专注于手术领域的特定知识，避免了传统的微调方法。通过冻结图像编码器，仅优化LoRA层和深度解码器，以更好地集成手术场景特征。

技术框架：Surgical-DINO的整体架构包括DINOv2的图像编码器、LoRA层和深度解码器。训练过程中，图像编码器保持不变，LoRA层和解码器则进行优化，以适应手术数据。

关键创新：本研究的主要创新在于引入低秩适应技术，将其应用于深度估计任务中，显著提高了模型在手术领域的适应性和性能。与传统方法相比，Surgical-DINO能够更有效地利用预训练模型的特征。

关键设计：在模型设计中，LoRA层的构建和参数设置至关重要。通过优化损失函数，使得模型能够更好地适应手术场景的特征，提升深度估计的准确性。

🖼️ 关键图片

fig_0
fig_1
fig_2

📊 实验亮点

在MICCAI挑战数据集SCARED上的实验结果显示，Surgical-DINO在内窥镜深度估计任务中显著优于所有现有的最先进模型，具体性能提升幅度未知，表明该方法在手术领域的有效性和适应性。

🎯 应用场景

Surgical-DINO的研究成果在内窥镜手术、3D重建、手术导航和增强现实可视化等领域具有广泛的应用潜力。通过提高深度估计的准确性，该模型能够为外科医生提供更可靠的视觉信息，进而提升手术的安全性和效率。未来，该方法还可扩展到其他医疗影像处理任务中。

📄 摘要（原文）

Purpose: Depth estimation in robotic surgery is vital in 3D reconstruction, surgical navigation and augmented reality visualization. Although the foundation model exhibits outstanding performance in many vision tasks, including depth estimation (e.g., DINOv2), recent works observed its limitations in medical and surgical domain-specific applications. This work presents a low-ranked adaptation (LoRA) of the foundation model for surgical depth estimation. Methods: We design a foundation model-based depth estimation method, referred to as Surgical-DINO, a low-rank adaptation of the DINOv2 for depth estimation in endoscopic surgery. We build LoRA layers and integrate them into DINO to adapt with surgery-specific domain knowledge instead of conventional fine-tuning. During training, we freeze the DINO image encoder, which shows excellent visual representation capacity, and only optimize the LoRA layers and depth decoder to integrate features from the surgical scene. Results: Our model is extensively validated on a MICCAI challenge dataset of SCARED, which is collected from da Vinci Xi endoscope surgery. We empirically show that Surgical-DINO significantly outperforms all the state-of-the-art models in endoscopic depth estimation tasks. The analysis with ablation studies has shown evidence of the remarkable effect of our LoRA layers and adaptation. Conclusion: Surgical-DINO shed some light on the successful adaptation of the foundation models into the surgical domain for depth estimation. There is clear evidence in the results that zero-shot prediction on pre-trained weights in computer vision datasets or naive fine-tuning is not sufficient to use the foundation model in the surgical domain directly. Code is available at https://github.com/BeileiCui/SurgicalDINO.