Semantic-Driven Scale and Spatial Selection for Efficient Cross-Modal Alignment in Referring Remote Sensing Image Segmentation

📄 arXiv: 2606.30244v1 📥 PDF

作者: Kun Li, Shengxi Gui, Francesco Nex, Michael Ying Yang

分类: cs.CV

发布日期: 2026-06-29

备注: Submitted

💡 一句话要点

提出S4ECA以解决遥感图像分割中的跨模态对齐问题

🎯 匹配领域: 支柱九：具身大模型 (Embodied Foundation Models)

关键词: 遥感图像分割 跨模态对齐 参数高效微调 多模态推理 深度学习

📋 核心要点

  1. 现有的RRSIS模型依赖全量微调，计算成本高且可能影响预训练模型的泛化能力。
  2. 提出S4ECA框架，通过双编码器适配器实现跨模态交互，采用可学习查询提取语义信息。
  3. 在RRSIS-D和RefSegRS数据集上，模型仅更新2.4%参数，取得了领先的性能，展示了高效性与精确性。

📝 摘要（中文）

遥感图像分割（RRSIS）旨在根据自然语言表达定位和分割遥感图像中的目标对象或区域。现有模型依赖于全量微调，计算成本高且可能削弱预训练模型的泛化能力。尽管参数高效微调（PET）提供了潜在的替代方案，但现有框架主要集中于单模态优化，未能捕捉多模态推理所需的复杂跨模态依赖。为了解决这些问题，我们提出了一种新框架S4ECA，通过参数高效适应实现有效的跨模态交互。该模型在RRSIS-D和RefSegRS数据集上仅更新2.4%的主干参数，展现出在复杂空中场景中的高效性和精确性。

🔬 方法详解

问题定义：本论文旨在解决遥感图像分割中的跨模态对齐问题。现有方法在全量微调过程中计算成本高，且可能导致预训练模型的特征表示失真。

核心思路：提出S4ECA框架，通过参数高效适应实现跨模态交互，设计双编码器适配器以提取语义信息和视觉特征。

技术框架：整体架构包括文本适配器和视觉适配器。文本适配器使用可学习查询从词级嵌入中提取语义代理，视觉适配器则通过多尺度密集提取器优化特征表示。

关键创新：最重要的创新在于引入语言引导的尺度和空间选择机制，动态强调相关视觉上下文，确保精确的跨模态对齐。

关键设计：模型仅更新2.4%的主干参数，采用多尺度特征提取和语言引导选择机制，确保高效性和精确性。损失函数和网络结构经过精心设计，以适应复杂的遥感场景。

🖼️ 关键图片

fig_0
fig_1
fig_2

📊 实验亮点

在RRSIS-D和RefSegRS数据集上，S4ECA模型仅更新2.4%的参数，取得了领先的性能，展示了在复杂空中场景中的高效性和精确性，显著优于现有基线方法。

🎯 应用场景

该研究在遥感图像分析、环境监测和城市规划等领域具有广泛的应用潜力。通过提高遥感图像分割的效率和精度，能够更好地支持决策制定和资源管理，推动智能城市和可持续发展目标的实现。

📄 摘要（原文）

Referring Remote Sensing Image Segmentation (RRSIS) seeks to localize and segment the target object or region specified by a natural language expression in a remote sensing image. While existing RRSIS models have benefited from large-scale foundation models, they predominantly rely on full fine-tuning. These approaches are computationally intensive and may weaken the generalization ability of pre-trained models, as extensive fine-tuning on significantly smaller downstream datasets can distort the well-structured feature representations learned during large-scale pre-training. Although Parameter-Efficient Tuning (PET) offers a potential alternative, existing PET frameworks primarily focus on single-modal optimization, failing to capture the complex cross-modal dependencies required for multimodal reasoning, while simultaneously struggling to bridge the substantial domain gap between natural scenes and aerial imagery. To address these limitations, we propose a novel framework, Semantic-driven Scale and Spatial Selection for Efficient Cross-modal Alignment (S4ECA), which enables effective and efficient cross-modal interaction through parameter-efficient adaptation. Specifically, we design a dual-encoder adapter architecture. The textual adapter employs learnable queries to distill highly semantic language proxies from word-level embeddings, facilitating early grounding. Simultaneously, the visual adapter refines hierarchical feature representations through a multi-scale dense extractor, followed by a language-guided scale and spatial selection mechanism that dynamically emphasizes relevant visual contexts, ensuring precise cross-modal alignment. By updating only 2.4% of the backbone parameters, our proposed model achieves state-of-the-art performance on the RRSIS-D and RefSegRS datasets, demonstrating superior efficiency and precision in complex aerial scenarios.