Mobile-Seed: Joint Semantic Segmentation and Boundary Detection for Mobile Robots

📄 arXiv: 2311.12651v3 📥 PDF

作者: Youqi Liao, Shuhao Kang, Jianping Li, Yang Liu, Yun Liu, Zhen Dong, Bisheng Yang, Xieyuanli Chen

分类: cs.CV, cs.AI, cs.RO

发布日期: 2023-11-21 (更新: 2024-03-11)

备注: Accepted by IEEE Robotics and Automation Letters (RA-L) 2024. Code, pre-trained models and additional results are available at https://whu-usi3dv.github.io/Mobile-Seed/

DOI: 10.1109/LRA.2024.3373235

🔗 代码/项目: PROJECT_PAGE

💡 一句话要点

提出Mobile-Seed以解决移动机器人语义分割与边界检测问题

🎯 匹配领域: 支柱一：机器人控制 (Robot Control) 支柱三：空间感知与语义 (Perception & Semantics)

关键词: 语义分割 边界检测 移动机器人 深度学习 实时推理 轻量级模型 多任务学习

📋 核心要点

  1. 现有方法多聚焦于轻量级语义分割，忽视了边界检测的重要性，导致边界信息提取不足。
  2. 提出Mobile-Seed框架，通过双通道编码器和主动融合解码器，实现语义分割与边界检测的联合学习。
  3. 在Cityscapes数据集上，Mobile-Seed在mIoU和mF-score上分别提升2.2和4.2个百分点，且推理速度达到23.9 FPS。

📝 摘要（中文）

精确快速地划定清晰边界和稳健的语义信息对于机器人抓取、实时语义映射和在线传感器校准等任务至关重要。尽管边界检测与语义分割是互补的任务，但现有研究多集中于轻量级语义分割模型，忽视了边界检测的重要性。本文提出了Mobile-Seed，一个轻量级的双任务框架，旨在同时进行语义分割和边界检测。该框架采用双通道编码器、主动融合解码器（AFD）和双任务正则化方法，能够有效提升语义分割性能并准确定位物体边界。实验结果表明，Mobile-Seed在Cityscapes数据集上相较于现有最先进方法在mIoU上提升了2.2个百分点，在mF-score上提升了4.2个百分点，同时在RTX 2080 Ti GPU上以1024x2048分辨率实现23.9帧每秒的在线推理速度。

🔬 方法详解

问题定义：本文旨在解决移动机器人在执行任务时对语义分割与边界检测的需求，现有方法往往忽视边界检测的重要性，导致语义信息不够准确。

核心思路：Mobile-Seed框架通过双通道编码器分别提取语义信息和边界信息，结合主动融合解码器动态调整信息融合，提升整体性能。

技术框架：整体架构包括双通道编码器、主动融合解码器（AFD）和双任务正则化模块。编码器一部分专注于类别感知的语义信息，另一部分则从多尺度特征中提取边界信息。

关键创新：引入主动融合解码器（AFD），通过学习通道间关系动态调整语义与边界信息的融合权重，显著提升了边界检测的准确性。

关键设计：采用双任务正则化损失函数，以减轻双任务学习中的冲突，并实现深度多样性监督，确保模型在学习过程中保持稳定性。实验中使用的网络结构和参数设置经过精心设计，以适应实时推理的需求。

🖼️ 关键图片

fig_0
fig_1
fig_2

📊 实验亮点

Mobile-Seed在Cityscapes数据集上相较于最先进的基线方法，mIoU提升了2.2个百分点，mF-score提升了4.2个百分点，同时在RTX 2080 Ti GPU上以1024x2048分辨率实现了23.9帧每秒的在线推理速度，显示出其优越的性能和实时处理能力。

🎯 应用场景

该研究的潜在应用领域包括移动机器人在复杂环境中的自主导航、物体抓取与操作、实时环境建模等。通过提升语义分割与边界检测的准确性，Mobile-Seed能够显著增强机器人在动态环境中的适应能力，推动智能机器人技术的发展。

📄 摘要（原文）

Precise and rapid delineation of sharp boundaries and robust semantics is essential for numerous downstream robotic tasks, such as robot grasping and manipulation, real-time semantic mapping, and online sensor calibration performed on edge computing units. Although boundary detection and semantic segmentation are complementary tasks, most studies focus on lightweight models for semantic segmentation but overlook the critical role of boundary detection. In this work, we introduce Mobile-Seed, a lightweight, dual-task framework tailored for simultaneous semantic segmentation and boundary detection. Our framework features a two-stream encoder, an active fusion decoder (AFD) and a dual-task regularization approach. The encoder is divided into two pathways: one captures category-aware semantic information, while the other discerns boundaries from multi-scale features. The AFD module dynamically adapts the fusion of semantic and boundary information by learning channel-wise relationships, allowing for precise weight assignment of each channel. Furthermore, we introduce a regularization loss to mitigate the conflicts in dual-task learning and deep diversity supervision. Compared to existing methods, the proposed Mobile-Seed offers a lightweight framework to simultaneously improve semantic segmentation performance and accurately locate object boundaries. Experiments on the Cityscapes dataset have shown that Mobile-Seed achieves notable improvement over the state-of-the-art (SOTA) baseline by 2.2 percentage points (pp) in mIoU and 4.2 pp in mF-score, while maintaining an online inference speed of 23.9 frames-per-second (FPS) with 1024x2048 resolution input on an RTX 2080 Ti GPU. Additional experiments on CamVid and PASCAL Context datasets confirm our method's generalizability. Code and additional results are publicly available at https://whu-usi3dv.github.io/Mobile-Seed/.