TAMI: Temporally Aligned, Missingness-Aware, and Interpretable Multimodal Fusion for Mental Health Assessment in Older Adults with Mild Cognitive Impairment

📄 arXiv: 2608.30857v1 📥 PDF

作者: Merna Bibars, Bolaji Omofojoye, Allan I. Levey, Rachel Hershenberg, Gari D. Clifford, Hyeokhyen Kwon

分类: cs.CV, cs.AI

发布日期: 2026-08-31

备注: 19 pages, submitted to IEEE Transactions on Affective Computing

💡 一句话要点

提出TAMI框架以解决老年人轻度认知障碍的心理健康评估问题

🎯 匹配领域: 支柱九：具身大模型 (Embodied Foundation Models)

关键词: 多模态融合 心理健康评估 时间对齐 缺失性感知 可解释性

📋 核心要点

  1. 现有多模态分析方法在处理老年轻度认知障碍患者的心理健康评估时存在时间错位、缺失值处理不当和缺乏细粒度解释等问题。
  2. 本文提出的TAMI框架通过对齐多模态特征、编码缺失性并基于问题上下文进行融合，解决了上述问题。
  3. 实验结果表明，TAMI在抑郁和焦虑的AUROC得分分别为0.68和0.69，且多模态特征的时间对齐显著提升了性能。

📝 摘要（中文）

老年轻度认知障碍患者的抑郁和焦虑常因医疗资源有限而被低估。现有的多模态分析方法存在三大局限：未能校正不同分辨率下的时间错位、缺失值处理不当以及缺乏对预测结果的细粒度解释。为此，本文提出了一种时序对齐、缺失感知和可解释的多模态融合框架（TAMI），能够在共享时间线上对语音、语言、面部和生理特征进行对齐，编码时间上的缺失性，并基于问题上下文进行融合。在对49名老年轻度认知障碍患者的访谈中，TAMI在抑郁和焦虑的接收者操作特征曲线下面积（AUROC）分别达到了0.68和0.69，显示出显著的性能提升。

🔬 方法详解

问题定义：本文旨在解决老年轻度认知障碍患者心理健康评估中存在的多模态特征时间错位、缺失值处理不当及缺乏细粒度解释等问题。现有方法未能有效处理这些挑战，导致评估结果的准确性和可解释性不足。

核心思路：TAMI框架通过在共享时间线上对齐语音、语言、面部和生理特征，编码时间上的缺失性，并结合问题上下文进行特征融合，从而提升评估的准确性和可解释性。

技术框架：TAMI的整体架构包括特征提取、时间对齐、缺失性编码和基于上下文的融合四个主要模块。特征提取模块从不同模态中提取相关特征，时间对齐模块确保这些特征在时间上同步，缺失性编码模块处理模态缺失情况，而融合模块则结合上下文信息进行最终的特征融合。

关键创新：TAMI的核心创新在于其对多模态特征的细粒度时间对齐和缺失性感知处理，这与现有方法的处理方式有本质区别，能够有效减少虚假跨模态关联。

关键设计：在模型设计中，TAMI采用了特定的损失函数来优化时间对齐效果，并在缺失性编码中引入了模态级别的缺失性标记。此外，网络结构设计上，TAMI结合了多层次的特征提取和融合策略，以增强模型的表达能力。

🖼️ 关键图片

fig_0
fig_1
fig_2

📊 实验亮点

TAMI框架在对49名老年轻度认知障碍患者的访谈中，抑郁和焦虑的AUROC得分分别为0.68和0.69，显示出显著的性能提升。特别是多模态特征的时间对齐带来了超过0.1的性能增益，表明该方法在心理健康评估中的有效性。

🎯 应用场景

该研究的潜在应用场景包括老年人心理健康的远程评估和干预，尤其是在资源有限的环境中。通过优化访谈协议，TAMI框架能够为老年轻度认知障碍患者提供更为精准的心理健康筛查，进而推动相关医疗服务的普及与提升。

📄 摘要（原文）

Depression and anxiety in older adults with Mild Cognitive Impairment (MCI) are frequently underdiagnosed due to limited access to care. Multimodal analysis of remote clinical interviews is a scalable screening approach, but existing methods have three limitations. First, they do not correct temporal misalignment across multimodal features extracted at different resolutions, inducing spurious cross-modal associations. Second, remote recordings exhibit uneven modality dropout, but missing values are often zero-filled, making them indistinguishable from valid near-zero measurements. Finally, they do not jointly attribute predictions to modalities, questions, and interview moments, limiting fine-grained clinical interpretation. We propose a Temporally-Aligned, Missingness-Aware, Interpretable (TAMI) multimodal fusion framework. TAMI aligns speech, language, facial, and physiological features within question-answer segments on a shared timeline, encodes modality-level missingness over time, and conditions fusion on question context. In interviews with 49 older adults with MCI, TAMI achieved area under the receiver operating characteristic curve (AUROC) scores of 0.68 (depression) and 0.69 (anxiety). Fine-grained temporal alignment of multimodal features produced the largest performance gain ($Δ{\geq}0.1$). Multi-level interpretability analysis revealed that depression classification relied on eyegaze and open-ended questions, while anxiety classification depended on eyegaze and head pose, with attribution uniformly distributed across questions. Using only responses to the open-ended questions (5.1min), the depression model achieved an AUROC score of 0.67, which was not significantly different from using the full interview (19min) ($p>0.05$). Our findings support designing interview protocols centered on open-ended questions for depression screening in older adults with MCI.