Omni-DuplexEval: Evaluating Real-time Duplex Omni-modal Interaction

📄 arXiv: 2605.17360 📥 PDF

作者: Chaoqun He, Mingyang Xiang, Yingjing Xu, Bokai Xu, Junbo Cui, Jie Zhou, Yuan Yao, Lijie Wen

分类: cs.CV

发布日期: 2026-07-05

💡 一句话要点

提出Omni-DuplexEval以解决实时双向多模态交互评估问题

🎯 匹配领域: 支柱九：具身大模型 (Embodied Foundation Models)

关键词: 实时交互 多模态评估 大型语言模型 自动评估 视频理解 人机交互

📋 核心要点

  1. 现有多模态大型语言模型在实时双向交互评估上存在不足，主要集中在离线处理，无法适应动态输入。
  2. 本文提出Omni-DuplexEval基准，系统评估实时双向交互能力，包含实时描述和主动提醒两个场景。
  3. 实验表明，当前最优模型在Omni-DuplexEval基准上仅得39.6%，主动提醒任务得分更低，揭示了模型在响应时机和内容生成上的挑战。

📝 摘要（中文）

实时双向交互对于在真实场景中运行的多模态人工智能系统至关重要，现有的大型多模态语言模型（MLLMs）大多在离线环境中进行评估，无法满足实时需求。为此，本文提出了Omni-DuplexEval，一个系统化评估实时双向交互的基准，包含实时描述和主动提醒两个场景，涵盖660个视频及其精确的时间元数据。我们还引入了一种基于LLM-as-a-Judge的自动评估框架，能够通过时间戳感知和序列推理对响应内容和时机进行联合评估。实验结果显示，现有模型在该基准上的表现存在显著局限性，最佳模型仅取得39.6%的总体得分，主动提醒任务得分更低，仅为20.0%。

🔬 方法详解

问题定义：本文旨在解决现有多模态大型语言模型在实时双向交互评估中的不足，尤其是缺乏系统化的基准和自动评估方法。现有方法主要在离线环境中评估，无法有效处理实时输入和响应。

核心思路：提出Omni-DuplexEval基准，通过实时描述和主动提醒两个场景，系统评估模型在动态环境中的交互能力。设计自动评估框架，利用LLM作为评判者，结合时间戳感知和序列推理，提升评估的准确性和一致性。

技术框架：Omni-DuplexEval基准包含660个视频，提供精细的人类标注标签和时间元数据。评估流程分为两个主要模块：实时描述模块和主动提醒模块，分别针对生成连续响应和识别重要事件进行评估。

关键创新：引入了基于LLM的自动评估框架，能够同时评估响应内容与时机的对齐性，显著提升了评估的系统性和准确性。这一方法与传统的离线评估方法有本质区别。

关键设计：在评估过程中，采用了时间戳感知机制和序列推理方法，以确保模型在生成响应时能够考虑到时间因素和上下文信息。

🖼️ 关键图片

fig_0
fig_1
fig_2

📊 实验亮点

实验结果显示，当前最优的多模态大型语言模型在Omni-DuplexEval基准上的总体得分仅为39.6%，而在主动提醒任务上得分更低，仅为20.0%。这些结果揭示了模型在平衡响应时机与内容生成方面的显著挑战。

🎯 应用场景

该研究的潜在应用领域包括智能助手、视频监控分析和人机交互等场景。通过提升多模态AI系统的实时交互能力，能够更好地满足用户需求，增强系统的实用性和响应性，未来可能推动相关技术的广泛应用。

📄 摘要（原文）

Real-time duplex interaction is essential for multimodal AI systems operating in real-world scenarios, where models must continuously process streaming inputs and respond at appropriate moments. However, most existing multimodal large language models (MLLMs) are evaluated in offline settings, where the entire video input is processed before any response is generated. While recent work has started to explore real-time duplex MLLMs, there is still no comprehensive benchmark or automatic evaluation method for this setting. To address this gap, we propose Omni-DuplexEval, a benchmark for systematically evaluating real-time duplex interaction. The benchmark consists of two complementary scenarios: (1) Real-Time Description, which evaluates the ability to generate continuous, time-aligned responses that track evolving multimodal inputs, and (2) Proactive Reminder, which evaluates the ability to identify salient events and respond at appropriate moments. Omni-DuplexEval contains 660 videos with fine-grained, human-annotated labels and precise temporal metadata, spanning 9 tasks grounded in real-world scenarios, where all questions are formulated as open-ended queries. We further introduce an automatic evaluation framework based on LLM-as-a-Judge, which enables systematic assessment by jointly evaluating response-content alignment and response timing through timestamp-aware and sequential reasoning, achieving strong alignment with human judgments. Experiments on state-of-the-art duplex MLLMs reveal substantial limitations. The best-performing model achieves only 39.6% overall, while scoring only 20.0% on Proactive Reminder. Our analysis identifies two key challenges: models struggle to balance timely responses with coherent, holistic content generation, and they often fail to determine both when to respond and what to produce. We hope our work facilitates further progress in MLLMs.