Omni-InteractScope: Benchmarking Proactive and Agentic Interaction in Continuous Omnimodal Streams

📄 arXiv: 2610.35439v1 📥 PDF

作者: Yinsong Yan, Ling Wang, Yuxuan Wang, Yifan Yang, Ruiyang Xu, Junming Lin, Qi Chen, Meng Gao, Minghao Han, Yunfei Chu, Qize Yang, Xize Cheng, Jin Xu, Jibin Wu

分类: cs.AI, cs.CV, cs.HC

发布日期: 2026-10-26

💡 一句话要点

提出Omni-InteractScope以解决Omni-LLMs在连续多模态流中的交互问题

🎯 匹配领域: 支柱九：具身大模型 (Embodied Foundation Models)

关键词: 全模态大语言模型 交互评估 多模态流 工具调用 用户需求 实时反馈 智能助手

📋 核心要点

  1. 现有的Omni-LLMs在连接视听流证据与用户需求方面存在显著不足，尤其是在及时干预和响应的可靠性上。
  2. 本文提出Omni-InteractScope基准，通过系统性评估1,479个交互，旨在提高Omni-LLMs在多模态流中的交互能力。
  3. 实验结果显示，12个评估模型在工具调用的及时性和无查询提醒的覆盖率上表现不佳，提示了未来改进的方向。

📝 摘要（中文）

随着全模态大语言模型（Omni-LLMs）在连续视听流中辅助用户的期望增加，现有评估方法对这些模型如何可靠地连接流证据与用户需求的洞察有限。本文提出了Omni-InteractScope基准，包含1,479个人工标注的交互和1,790个时间目标，涵盖事件触发、干预类型和用户需求。实验结果揭示了当前Omni-LLMs的三大关键局限性：工具调用的及时性不可靠、无查询提醒覆盖不足以及端到端指导覆盖有限。这些发现促使开发和评估能够及时行动的流媒体助手。

🔬 方法详解

问题定义：本文旨在解决现有Omni-LLMs在连续多模态流中对用户需求的及时响应和干预能力不足的问题。现有方法在工具调用的及时性和无查询提醒的覆盖率上存在明显短板。

核心思路：提出Omni-InteractScope基准，通过系统化的交互评估，连接输入模态、事件触发、干预类型和用户需求，以提升模型的交互能力和响应准确性。

技术框架：整体架构包括数据收集、交互标注、模型评估和结果分析四个主要模块。通过对1,479个交互的分析，评估模型在不同任务组中的表现。

关键创新：最重要的创新在于构建了一个全面的交互基准，涵盖了事件触发和用户需求的多样性，填补了现有评估方法的空白。

关键设计：在实验中，采用了多种评估指标，如工具调用的及时性和无查询提醒的覆盖率，确保了评估的全面性和准确性。

🖼️ 关键图片

fig_0
fig_1
fig_2

📊 实验亮点

实验结果显示，12个评估模型在工具调用的及时性上，最多仅有40%的目标能够按时响应，且联合时间和工具的准确率最高仅为32.0%。此外，无查询提醒的及时参考目标召回率从未超过23.0%，显示出当前模型在这些方面的显著不足。

🎯 应用场景

该研究的潜在应用领域包括智能助手、视频监控、在线教育等场景，能够提升用户与系统的交互体验，促进实时反馈和指导的实现。未来，随着技术的进步，Omni-LLMs有望在更多复杂应用中发挥重要作用。

📄 摘要（原文）

Omnimodal large language models (Omni-LLMs) are increasingly expected to assist users while observing continuous audiovisual streams. However, existing evaluations provide limited insight into how reliably these models connect stream evidence and user needs to timely, sustained interventions. We introduce \textbf{Omni-InteractScope}, a benchmark of 1,479 human-annotated, cross-reviewed interactions with 1,790 temporal targets across eight task groups. It links input modalities, event triggers, intervention types, and user needs, covering event-timed tool invocation and query-free assistance alongside notifications, warnings, state tracking, and continuous responses. Our experimental results on the proposed benchmark reveal three critical limitations of current Omni-LLMs: (1) \textbf{Timely tool invocation remains unreliable}: all 12 evaluated models produce an on-time first response for at most 40\% of tool-invocation targets, and joint timing-and-tool accuracy reaches at most 32.0\%. (2) \textbf{Query-free reminders remain poorly covered}: all evaluated models achieve lower timely reference-target recall on query-free reminders than on explicit questions, measured on their respective target sets. Reminder recall never exceeds 23.0\%. (3) \textbf{End-to-end guidance coverage remains limited}: nine models cover none of the 26 guidance sequences in full, and the remaining three cover only one to three sequences each. Targeted controls further show that tool-invocation scores alone do not establish event grounding and that reminder coverage depends on response history and frequency. These findings motivate developing and evaluating streaming assistants for timely action, initiative under general user needs, and coverage of entire response sequences.