Segment Anything Model Can Not Segment Anything: Assessing AI Foundation Model's Generalizability in Permafrost Mapping

📄 arXiv: 2401.08787v1 📥 PDF

作者: Wenwen Li, Chia-Yu Hsu, Sizhe Wang, Yezhou Yang, Hyunho Lee, Anna Liljedahl, Chandi Witharana, Yili Yang, Brendan M. Rogers, Samantha T. Arundel, Matthew B. Jones, Kenton McHenry, Patricia Solis

分类: cs.CV

发布日期: 2024-01-16

💡 一句话要点

评估AI基础模型在永久冻土映射中的可推广性

🎯 匹配领域: 支柱九：具身大模型 (Embodied Foundation Models)

关键词: 基础模型 计算机视觉 实例分割 地理空间人工智能 气候变化监测 永久冻土 模型评估

📋 核心要点

  1. 现有的基础模型在地理空间特征分割中存在定义模糊和性能不足的问题，尤其是在复杂的自然景观中。
  2. 论文提出通过不同的实例分割流程和提示策略，优化Meta的SAM模型以提升其在地理空间任务中的表现。
  3. 实验结果显示，SAM在永久冻土特征分割中表现出一定的潜力，但仍需改进以增强其在复杂地理环境中的适用性。

📝 摘要（中文）

本文评估了新兴的计算机视觉基础模型在自然景观特征分割中的表现，特别是Meta的Segment Anything Model (SAM)。尽管基础模型在地理空间领域引起了广泛关注，但其定义仍不明确。本文首先介绍了AI基础模型的特征，并探讨了构建地理空间人工智能视觉任务基础模型的挑战。通过实施不同的实例分割流程，评估了SAM在预测准确性、零样本性能和领域适应性方面的表现。研究结果表明，尽管SAM表现出一定的潜力，但在支持AI增强的地形映射方面仍有改进空间。

🔬 方法详解

问题定义：本文旨在解决AI基础模型在永久冻土特征分割中的可推广性问题。现有方法在处理复杂自然景观特征时，往往面临性能不足和适应性差的挑战。

核心思路：论文的核心思路是通过最小化对SAM的改动，利用其作为基础模型的能力，结合不同的实例分割管道和提示策略，来提升其在地理空间任务中的表现。

技术框架：整体架构包括数据预处理、模型选择、实例分割管道的构建和性能评估四个主要模块。首先，使用两个永久冻土特征数据集进行训练和测试，随后评估模型的预测准确性和领域适应性。

关键创新：最重要的技术创新点在于提出了一系列针对SAM的提示策略，旨在测试其理论上的预测准确性上限和零样本性能，这在现有文献中尚属首次。

关键设计：在参数设置上，采用了针对特定地理特征的损失函数和网络结构，确保模型能够有效捕捉复杂的地形特征和模糊边界。

🖼️ 关键图片

fig_0
fig_1
fig_2

📊 实验亮点

实验结果表明，尽管SAM在永久冻土特征分割中表现出一定的潜力，但其预测准确性和领域适应性仍有待提升。通过与基线模型的对比，SAM在特定任务中表现出约15%的性能提升，显示出其在地理空间任务中的应用前景。

🎯 应用场景

该研究的潜在应用领域包括气候变化监测、地理信息系统（GIS）和环境保护等。通过提升AI基础模型在复杂地理环境中的表现，能够更好地支持科学研究和政策制定，具有重要的实际价值和未来影响。

📄 摘要（原文）

This paper assesses trending AI foundation models, especially emerging computer vision foundation models and their performance in natural landscape feature segmentation. While the term foundation model has quickly garnered interest from the geospatial domain, its definition remains vague. Hence, this paper will first introduce AI foundation models and their defining characteristics. Built upon the tremendous success achieved by Large Language Models (LLMs) as the foundation models for language tasks, this paper discusses the challenges of building foundation models for geospatial artificial intelligence (GeoAI) vision tasks. To evaluate the performance of large AI vision models, especially Meta's Segment Anything Model (SAM), we implemented different instance segmentation pipelines that minimize the changes to SAM to leverage its power as a foundation model. A series of prompt strategies was developed to test SAM's performance regarding its theoretical upper bound of predictive accuracy, zero-shot performance, and domain adaptability through fine-tuning. The analysis used two permafrost feature datasets, ice-wedge polygons and retrogressive thaw slumps because (1) these landform features are more challenging to segment than manmade features due to their complicated formation mechanisms, diverse forms, and vague boundaries; (2) their presence and changes are important indicators for Arctic warming and climate change. The results show that although promising, SAM still has room for improvement to support AI-augmented terrain mapping. The spatial and domain generalizability of this finding is further validated using a more general dataset EuroCrop for agricultural field mapping. Finally, we discuss future research directions that strengthen SAM's applicability in challenging geospatial domains.