Multimodal Prompt Learning with Dual-Granularity Contrastive Alignment for Preoperative Prediction of Lung Adenocarcinoma Invasiveness Using 3D CT

📄 arXiv: 2611.03109v1 📥 PDF

作者: Qing Huang, Xiaotong Tan, Lintong Yao, Wufeng Xue, Ruobing Huang, Dong Ni, Hao Wu, Tian Xiao, Jun Cheng

分类: cs.CV, cs.AI

发布日期: 2026-11-02

💡 一句话要点

提出多模态提示学习框架以解决肺腺癌侵袭性预测问题

🎯 匹配领域: 支柱九：具身大模型 (Embodied Foundation Models)

关键词: 肺腺癌 侵袭性预测 多模态学习 3D CT 对齐机制 RadBERT 深度学习

📋 核心要点

  1. 现有方法在CT影像中难以区分侵袭性腺癌与前IAD病变，影响临床决策。
  2. 提出的框架结合了全局和局部对齐，通过多模态提示学习提升分类性能。
  3. 实验结果显示，该方法在多个指标上超越了传统图像分析方法，验证了其有效性。

📝 摘要（中文）

准确的肺腺癌侵袭性预测对手术规划和个性化治疗至关重要。然而，CT影像的重叠特征使得区分侵袭性腺癌（IAD）与前IAD病变变得困难。本文提出了一种基于3D CT的多模态提示学习框架，利用双粒度图像-文本对齐进行IAD分类。通过3D ResNet-CBAM编码器提取全局和空间视觉特征，并使用冻结的RadBERT骨干网络，结合报告级和关键词级的可学习提示进行文本编码。实验结果表明，该方法在测试集上达到了87.92%的准确率和0.9499的AUC，显著优于仅使用图像的基线方法。

🔬 方法详解

问题定义：本文旨在解决肺腺癌侵袭性预测中的分类困难，现有方法在CT影像中难以有效区分IAD与前IAD病变，导致临床决策受限。

核心思路：提出的多模态提示学习框架通过双粒度的图像-文本对齐，结合全局和局部特征，增强了对侵袭性特征的捕捉能力，从而提高分类准确性。

技术框架：整体架构包括3D ResNet-CBAM编码器用于提取视觉特征，RadBERT骨干网络用于文本编码，结合全局对齐和局部对齐模块进行优化。

关键创新：最重要的创新在于双粒度对齐机制，通过全局和局部特征的结合，显著提升了CT影像的语义理解能力，与传统方法相比具有本质区别。

关键设计：采用可学习的提示初始化，基于真实临床报告的语义表示，损失函数设计上结合了分类和对齐目标，确保了模型的有效训练。

📊 实验亮点

在包含1,593个病理确认结节的实验中，提出的方法在测试集上达到了87.92%的准确率、87.78%的F1分数和0.9499的AUC，显著优于仅使用图像的Res-CBAM基线（AUC为0.9297，p = 0.0078），验证了双粒度对齐的有效性。

🎯 应用场景

该研究的潜在应用领域包括肺癌的早期诊断和个性化治疗方案的制定。通过提高对腺癌侵袭性的预测能力，可以帮助医生在手术前做出更为精准的决策，从而改善患者的预后。未来，该方法可能扩展到其他类型肿瘤的影像分析中。

📄 摘要（原文）

Accurate preoperative prediction of lung adenocarcinoma invasiveness is important for surgical planning and individualized treatment. However, overlapping CT appearances make it challenging to distinguish invasive adenocarcinoma (IAD) from Pre-IAD lesions. We propose a multimodal prompt-learning framework with dual-granularity image-text alignment for 3D CT-based IAD classification. A 3D ResNet-CBAM encoder extracts global and spatial visual features. A frozen RadBERT backbone is adapted with separate learnable prompts for report-level and keyword-level text encoding. These prompts are initialized using semantic representations derived from real-world clinical reports. Global contrastive alignment connects whole-volume CT representations with reports generated from clinician-annotated radiological attributes to provide overall semantic supervision. A multiple instance learning-based local alignment module associates spatial CT features with vascular-focused radiological keywords, to capture local imaging characteristics potentially associated with tumor invasion. The framework jointly optimizes classification and alignment objectives, while inference requires only CT images. Experiments included 1,593 pathologically confirmed nodules. On the test set, the proposed method achieved an accuracy of 87.92%, F1-score of 87.78%, and AUC of 0.9499. Its AUC significantly exceeded that of the image-only Res-CBAM baseline with the same visual backbone (0.9297; DeLong test, p = 0.0078). Ablation analyses supported the complementary contributions of global and local alignment and the benefit of report-based prompt initialization. Qualitative visualizations showed activation within nodules and adjacent vascular structures. These findings suggest that dual-granularity radiological supervision can enhance CT-based invasiveness prediction while preserving a CT-only inference workflow.