SemanticXR: Low Power and Real-time Queryable Semantic Mapping with an Object-Level Device-Cloud Architecture
作者: Rahul Singh, Devdeep Ray, Connor Smith, Sarita Adve
分类: cs.DC, cs.CV, cs.RO
发布日期: 2026-06-11
💡 一句话要点
提出SemanticXR以解决移动XR设备的语义映射问题
🎯 匹配领域: 支柱三：空间感知与语义 (Perception & Semantics)
关键词: 语义映射 增强现实 设备-云架构 低功耗 实时查询 对象级处理 空间物体搜索
📋 核心要点
- 现有语义映射方法计算密集，无法在移动XR设备上实时运行，且依赖于服务器资源。
- SemanticXR通过将语义对象作为第一类单元，优化了设备与云端的通信和执行，提升了系统的实时性和低功耗特性。
- 实验结果显示，SemanticXR在服务器端映射延迟上提高了2.2倍，设备端查询延迟低于100毫秒，且功耗增加仅为2%。
📝 摘要（中文）
语义映射是增强现实（XR）应用中实现交互的核心服务，然而现有方法计算密集且依赖服务器资源。本文提出SemanticXR，这是首个在XR功耗、带宽和内存限制下实现实时开放词汇语义映射和查询的设备-云系统。通过将语义可识别对象提升为通信、执行和内存的第一类单元，SemanticXR在服务器端通过对象级并行和几何降采样提高映射延迟，同时在设备端通过对象级稀疏本地地图实现网络鲁棒查询。实验表明，SemanticXR在相同语义质量下，服务器端映射延迟提高了2.2倍，设备端查询延迟低于100毫秒，且仅增加2%的设备功耗。
🔬 方法详解
问题定义：本文旨在解决移动XR设备上实时语义映射的挑战，现有方法计算密集且依赖于服务器资源，无法满足低功耗和实时性的需求。
核心思路：SemanticXR的核心思想是将语义可识别对象提升为通信、执行和内存的第一类单元，从而在设备和服务器之间实现高效的资源管理和数据传输。
技术框架：SemanticXR的整体架构包括设备端的对象级稀疏本地地图和服务器端的对象级并行处理。设备端负责实时查询和更新，服务器端则进行深度映射和几何降采样以优化延迟。
关键创新：SemanticXR的主要创新在于对象级的资源配置与质量权衡，允许应用根据需求动态调整映射策略，与现有方法相比，显著提高了映射效率和查询速度。
关键设计：系统设计中，采用了增量更新和更新优先级策略以优化内存和带宽使用，深度映射的协同设计确保了上行带宽保持在2.5 Mbps以下，设备端支持最多10,000个对象的查询。
🖼️ 关键图片
📊 实验亮点
在与相同感知模型的设备-云基线对比中，SemanticXR在服务器端映射延迟上提高了2.2倍，且在设备端支持低于100毫秒的查询延迟，能够处理多达10,000个对象，功耗增加仅为2%。
🎯 应用场景
SemanticXR的研究成果在增强现实、人工智能助手和空间物体搜索等领域具有广泛的应用潜力。通过实现低功耗和实时的语义映射，能够提升用户体验，支持更复杂的交互场景，推动XR技术的进一步发展。
📄 摘要（原文）
Semantic mapping is a core service that enables grounded interactions in emerging Extended Reality (XR) applications such as AI assistants and spatial object search. Deploying this capability on mobile XR devices requires a system that is open-vocabulary, real-time, and low-power. Existing approaches are compute-intensive and assume server-class resources. Cloud offloading offers a practical path, but no existing system splits semantic mapping across the device-cloud boundary or manages its communication, execution, and memory footprint. We present SemanticXR, the first device-cloud system for real-time, open-vocabulary semantic mapping and querying under XR power, bandwidth, and memory constraints. Our key insight is to elevate semantically identifiable objects to first-class units of communication, execution, and memory across the device and server. On the server, object-level parallelism and geometry downsampling improve mapping latency, while object-level depth-mapping co-design reduces upstream bandwidth. On the device, an object-level sparse local map with incremental updates and update prioritization enables network-robust querying with bounded memory and downstream bandwidth. Object-level configurable resource usage vs. quality trade-offs let applications and the system adapt mapping to application requirements and operating conditions, respectively. Against a device-cloud baseline with the same perception models, object-level organization improves server-side mapping latency by 2.2X at equal semantic quality. Depth-mapping co-design maintains upstream bandwidth under 2.5 Mbps. On the device, SemanticXR sustains sub-100 ms query latency for up to 10,000 objects even under network drops, supports tens of thousands of objects within 500 MB, and scales downstream bandwidth with map changes, not total scene size. The system adds only 2% device power during normal operation.