M3BAT: Unsupervised Domain Adaptation for Multimodal Mobile Sensing with Multi-Branch Adversarial Training

📄 arXiv: 2404.17391v1 📥 PDF

作者: Lakmal Meegahapola, Hamza Hassoune, Daniel Gatica-Perez

分类: cs.LG, cs.AI, cs.CY, cs.HC

发布日期: 2024-04-26

备注: Accepted at the Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies (IMWUT). Paper will be presented at ACM UbiComp 2024

DOI: 10.1145/3659591

💡 一句话要点

提出M3BAT以解决多模态移动传感中的无监督领域适应问题

🎯 匹配领域: 支柱二：RL算法与架构 (RL & Architecture) 支柱九：具身大模型 (Embodied Foundation Models)

关键词: 多模态传感 无监督领域适应 对抗训练 移动健康 数据分布转移

📋 核心要点

  1. 现有方法主要集中于单一模态数据，缺乏对多模态传感数据的无监督领域适应研究。
  2. 本文提出M3BAT，通过多分支对抗训练来有效处理多模态传感数据的分布转移问题。
  3. 在两个多模态移动传感数据集上进行的实验表明，该方法在未见领域上显著提高了模型性能。

📝 摘要（中文）

近年来，多模态移动传感在健康、行为和环境推断方面得到了广泛应用。然而，训练集与真实环境数据分布差异的问题严重阻碍了模型的实际部署。尽管在计算机视觉和自然语言处理领域对此问题进行了广泛研究，但在多模态传感数据的无监督领域适应方面的研究仍然较少。为此，本文通过对领域对抗神经网络（DANN）的深入实验，提出了一种名为M3BAT的新方法，利用多分支对抗训练来处理多模态传感数据的分布转移。实验结果表明，与直接在源领域训练的模型相比，该方法在分类任务上性能提高了最高12%的AUC，在回归任务上提高了0.13的MAE。

🔬 方法详解

问题定义：本文旨在解决多模态移动传感中的无监督领域适应问题，现有方法主要集中于单一模态，未能有效处理数据分布转移带来的挑战。

核心思路：提出M3BAT，通过多分支对抗训练来同时考虑多模态传感数据的特性，从而提高模型在不同领域的适应能力。

技术框架：M3BAT的整体架构包括多个分支，每个分支处理不同模态的数据，通过对抗训练机制来减少源领域与目标领域之间的分布差异。

关键创新：M3BAT的创新在于引入多分支结构，使得模型能够同时学习多种模态的数据特征，从而更好地应对领域转移问题，这与传统的单模态对抗训练方法有本质区别。

关键设计：在模型设计中，采用了特定的损失函数来平衡各个模态的学习，同时设置了适应性参数以优化训练过程，确保模型在不同任务上的表现。

🖼️ 关键图片

fig_0
fig_1
fig_2

📊 实验亮点

实验结果显示，M3BAT在分类任务上相较于源领域模型性能提升最高达12%的AUC，在回归任务上提升0.13的MAE，表明该方法在处理未见领域数据时的有效性和优势。

🎯 应用场景

该研究的潜在应用领域包括健康监测、行为分析和环境感知等多个场景，能够为智能手机和可穿戴设备提供更准确的推断能力。未来，M3BAT有望推动多模态传感技术在实际应用中的广泛部署，提升用户体验和数据分析的准确性。

📄 摘要（原文）

Over the years, multimodal mobile sensing has been used extensively for inferences regarding health and well being, behavior, and context. However, a significant challenge hindering the widespread deployment of such models in real world scenarios is the issue of distribution shift. This is the phenomenon where the distribution of data in the training set differs from the distribution of data in the real world, the deployment environment. While extensively explored in computer vision and natural language processing, and while prior research in mobile sensing briefly addresses this concern, current work primarily focuses on models dealing with a single modality of data, such as audio or accelerometer readings, and consequently, there is little research on unsupervised domain adaptation when dealing with multimodal sensor data. To address this gap, we did extensive experiments with domain adversarial neural networks (DANN) showing that they can effectively handle distribution shifts in multimodal sensor data. Moreover, we proposed a novel improvement over DANN, called M3BAT, unsupervised domain adaptation for multimodal mobile sensing with multi-branch adversarial training, to account for the multimodality of sensor data during domain adaptation with multiple branches. Through extensive experiments conducted on two multimodal mobile sensing datasets, three inference tasks, and 14 source-target domain pairs, including both regression and classification, we demonstrate that our approach performs effectively on unseen domains. Compared to directly deploying a model trained in the source domain to the target domain, the model shows performance increases up to 12% AUC (area under the receiver operating characteristics curves) on classification tasks, and up to 0.13 MAE (mean absolute error) on regression tasks.