YANAMI: A Large-Scale Multimodal Dataset and Benchmark for Multiturn Stateful Interaction

📄 arXiv: 2611.05280v1 📥 PDF

作者: Shengyu Wang, Tengyue Nong, Junming Liu, Haosong Peng, Siyu Li, Ruiqi Lei, Fushuo Huo, Yuhao Pan, Yunfeng Fan, Ding Wang, Nan Cheng, Haozhao Wang, Wenchao Xu

分类: cs.CV

发布日期: 2026-11-03

备注: Project page: https://rowletqwq.github.io/YANAMI

💡 一句话要点

提出YANAMI以解决多轮状态交互的多模态数据集不足问题

🎯 匹配领域: 支柱二：RL算法与架构 (RL & Architecture) 支柱九：具身大模型 (Embodied Foundation Models)

关键词: 多模态对话 状态交互 数据集 视觉理解 模型评估 图像编辑 多轮对话

📋 核心要点

  1. 现有的多模态数据集主要集中在单轮任务，缺乏对多轮状态交互的有效监督，限制了模型的能力。
  2. 本文提出了YANAMI数据集，包含400K多轮对话，要求模型在对话中持续跟踪和更新状态，解决了现有方法的不足。
  3. 实验结果显示，微调BAGEL模型在YANAMI-400K上，图像编辑、生成和多模态状态交互的性能分别提升了10.3%、2.0%和13.23%。

📝 摘要（中文）

统一多模态模型（UMMs）将视觉理解、生成和编辑整合在一个统一框架内。然而，现有的多模态数据集和基准主要集中在单轮任务上，缺乏对多轮状态交互的有效监督和评估。为此，本文引入了YANAMI，这是第一个大规模多模态对话数据集和基准，包含400K多轮对话，涵盖多种场景，要求模型在对话中持续跟踪和更新状态。实验表明，针对YANAMI-400K进行微调能显著提升图像编辑和多模态状态交互的性能，同时保持一般视觉理解能力。

🔬 方法详解

问题定义：本文旨在解决现有多模态数据集在多轮状态交互中的不足，特别是缺乏对对话状态的跟踪和更新能力。现有方法在多轮任务中往往无法有效维护和实现对话状态的演变。

核心思路：论文提出了YANAMI数据集，包含多种场景的多轮对话，要求模型在对话中持续跟踪、保存和修正多模态状态，从而实现更复杂的交互。

技术框架：YANAMI数据集分为两个部分：YANAMI-400K和YANAMI-Bench。前者包含400K多轮对话和超过2M图像，后者用于系统评估多模态状态交互，涵盖对话状态理解和视觉实现。

关键创新：YANAMI是首个针对多轮状态交互的大规模多模态对话数据集，填补了现有数据集在这一领域的空白，特别是在模型理解和实现对话状态方面的能力评估。

关键设计：在数据集构建中，设计了多种场景（如多图像融合、图像编辑等），并确保对话状态的演变能够被有效跟踪和更新，采用了适合多模态任务的损失函数和网络结构。

🖼️ 关键图片

fig_0
fig_1
fig_2

📊 实验亮点

实验结果显示，针对YANAMI-400K进行微调的BAGEL模型在图像编辑、生成和多模态状态交互方面的性能分别提升了10.3%、2.0%和13.23%。评估结果表明，现有的统一多模态模型在维护和实现演变状态方面仍存在显著挑战。

🎯 应用场景

该研究的潜在应用领域包括智能客服、虚拟助手和人机交互等场景，能够提升多模态对话系统的智能化水平和用户体验。未来，YANAMI数据集有望推动多模态模型在复杂交互任务中的发展，促进更自然的对话生成与理解。

📄 摘要（原文）

Unified Multimodal Models (UMMs) integrate visual understanding, generation, and editing within a single unified framework. However, existing unified multimodal datasets and benchmarks remain dominated by single-turn tasks. Even in recent multi-turn datasets, many turns can be solved independently without tracking information across turns. Consequently, they provide limited supervision and evaluation for multimodal stateful interaction, which requires models to understand, maintain, update, and faithfully realize evolving dialogue states across turns. To address this gap, we introduce YANAMI, the first large-scale multimodal dialogue dataset and benchmark for stateful interaction, comprising YANAMI-400K and YANAMI-Bench. YANAMI-400K contains 400K multi-turn multimodal dialogues, with more than 2M images and 2.6M dialogue turns. The dialogues span four representative scenarios: Multi-Image Fusion, Image Editing, World Modeling, and Visual Reasoning, and require models to continuously track, preserve, and revise evolving multimodal dialogue states across turns. YANAMI-Bench comprises 400 multi-turn dialogues for systematically evaluating multimodal stateful interaction. It jointly evaluates Dialogue State Understanding and Visual Realization, characterizing the gap between models' ability to understand evolving dialogue states and their ability to faithfully realize them in visual outputs. Experiments show that fine-tuning BAGEL on YANAMI-400K improves image editing, generation, and multimodal stateful interaction by 10.3%, 2.0%, and 13.23%, respectively, while preserving its general visual understanding capabilities. Evaluations on YANAMI-Bench further reveal that existing UMMs still struggle to maintain, update, and visually realize evolving multimodal dialogue states.