MIntRec2.0: A Large-scale Benchmark Dataset for Multimodal Intent Recognition and Out-of-scope Detection in Conversations
作者: Hanlei Zhang, Xin Wang, Hua Xu, Qianrui Zhou, Kai Gao, Jianhua Su, jinyue Zhao, Wenrui Li, Yanting Chen
分类: cs.MM, cs.CL
发布日期: 2024-03-16 (更新: 2024-06-28)
备注: Accepted by ICLR 2024, Long Paper; The abstract is slightly modified due to the length limitation
🔗 代码/项目: GITHUB
💡 一句话要点
提出MIntRec2.0以解决多模态意图识别和超出范围样本检测问题
🎯 匹配领域: 支柱九：具身大模型 (Embodied Foundation Models)
关键词: 多模态意图识别 超出范围样本检测 对话系统 数据集构建 人机交互
📋 核心要点
- 现有的多模态意图识别方法在规模和处理多轮对话中的超出范围样本方面存在显著不足。
- MIntRec2.0数据集通过提供大规模的多模态对话数据，增强了对人类意图的理解，并支持多轮对话的研究。
- 实验表明，尽管现有方法在融合非语言信息方面有所改进，但在上下文信息利用和超出范围样本检测上仍面临挑战。
📝 摘要（中文）
多模态意图识别面临重大挑战，需要结合真实场景中的非语言模态以增强对人类意图的理解。现有基准数据集规模有限，且在处理多轮对话中的超出范围样本时存在困难。本文介绍了MIntRec2.0，一个用于多方对话的多模态意图识别的大规模基准数据集，包含1,245个对话和15,040个样本，采用30个细粒度类别的新意图分类法进行标注。数据集中不仅包含9,304个范围内样本，还包括5,736个在多轮上下文中出现的超出范围样本。此外，论文提供了每个发言者的全面信息，增强了其在多方对话研究中的实用性。我们建立了一个通用框架，支持单轮和多轮对话数据的组织、模态特征提取、多模态融合，以及范围内分类和超出范围检测。
🔬 方法详解
问题定义：本论文旨在解决多模态意图识别中的数据集规模不足及超出范围样本检测困难的问题。现有方法在多轮对话中难以有效处理这些挑战。
核心思路：提出MIntRec2.0数据集，包含丰富的多模态对话样本，支持意图识别和超出范围样本的检测，旨在提升对人类意图的理解。
技术框架：整体架构包括数据组织、模态特征提取、多模态融合、范围内分类和超出范围检测等主要模块，形成一个完整的多模态意图识别流程。
关键创新：MIntRec2.0数据集的构建及其新意图分类法是本研究的核心创新，尤其是在处理多轮对话中的超出范围样本方面，与现有方法相比具有显著优势。
关键设计：在数据集构建中，采用了细粒度的意图分类，并提供了每个发言者的详细信息，增强了数据集的实用性和研究价值。
🖼️ 关键图片
📊 实验亮点
实验结果表明，尽管现有方法在融合非语言信息方面有所提升，但大型语言模型在意图理解任务中与人类评估者相比仍存在显著性能差距，强调了当前机器学习方法的局限性。
🎯 应用场景
MIntRec2.0数据集可广泛应用于人机对话系统、智能客服、语音助手等领域，提升这些系统在多模态意图识别和超出范围样本处理方面的能力，推动相关技术的进步与应用。
📄 摘要（原文）
Multimodal intent recognition poses significant challenges, requiring the incorporation of non-verbal modalities from real-world contexts to enhance the comprehension of human intentions. Existing benchmark datasets are limited in scale and suffer from difficulties in handling out-of-scope samples that arise in multi-turn conversational interactions. We introduce MIntRec2.0, a large-scale benchmark dataset for multimodal intent recognition in multi-party conversations. It contains 1,245 dialogues with 15,040 samples, each annotated within a new intent taxonomy of 30 fine-grained classes. Besides 9,304 in-scope samples, it also includes 5,736 out-of-scope samples appearing in multi-turn contexts, which naturally occur in real-world scenarios. Furthermore, we provide comprehensive information on the speakers in each utterance, enriching its utility for multi-party conversational research. We establish a general framework supporting the organization of single-turn and multi-turn dialogue data, modality feature extraction, multimodal fusion, as well as in-scope classification and out-of-scope detection. Evaluation benchmarks are built using classic multimodal fusion methods, ChatGPT, and human evaluators. While existing methods incorporating nonverbal information yield improvements, effectively leveraging context information and detecting out-of-scope samples remains a substantial challenge. Notably, large language models exhibit a significant performance gap compared to humans, highlighting the limitations of machine learning methods in the cognitive intent understanding task. We believe that MIntRec2.0 will serve as a valuable resource, providing a pioneering foundation for research in human-machine conversational interactions, and significantly facilitating related applications. The full dataset and codes are available at https://github.com/thuiar/MIntRec2.0.