Learning to Evolve Scenes: Reasoning about Human Activities with Scene Graphs

📄 arXiv: 2607.02425v1 📥 PDF

作者: Francesca Pistilli, Simone Alberto Peirone, Giuseppe Averta

分类: cs.CV

发布日期: 2026-07-02

备注: Project page at https://francescapistilli.github.io/GLEN

💡 一句话要点

提出SG-Ego和GLEN以解决人类活动理解问题

🎯 匹配领域: 支柱六：视频提取与匹配 (Video Extraction) 支柱九：具身大模型 (Embodied Foundation Models)

关键词: 时空场景图 人类活动理解 图神经网络 动态推理 多模态学习

📋 核心要点

  1. 现有方法多依赖隐式表示，缺乏对场景动态的结构化推理，限制了人类活动理解的深度。
  2. 本文提出SG-Ego注释集和GLEN模型，通过时空场景图实现对人类-环境互动的明确推理。
  3. GLEN在多个基准测试中表现优异，相较于原始视频基线，推理能力显著提升，尤其在长时间推理任务中表现突出。

📝 摘要（中文）

理解人类在与周围环境互动时的行为对于许多具身人工智能应用至关重要。第一人称视频在捕捉活动如何随时间重塑场景方面特别有效。然而，现有方法往往依赖隐式的视觉或语言对齐表示，忽视了对场景动态的结构化推理。为此，本文引入了SG-Ego，一个扩展Ego4D的大规模注释集，结合时空场景图，明确描述场景状态的时间演变。我们提出了GLEN，一个基于图的模型，能够对场景图序列进行推理，并与文本动作对齐。我们还提出了活动驱动的图编辑预测（A-GEF）问题，允许对场景变化进行明确推理。实验结果表明，GLEN在多个下游任务中表现优异，尤其在推理设置中超越了传统的多模态语言模型。

🔬 方法详解

问题定义：本文旨在解决现有方法在理解人类活动时对场景动态推理不足的问题。现有方法往往依赖隐式的视觉或语言表示，缺乏对场景变化的明确建模。

核心思路：论文提出了SG-Ego注释集，结合时空场景图，提供了明确的、可编辑的人类-环境互动表示。GLEN模型通过对场景图序列的推理，能够有效对齐文本动作并建模其时间演变。

技术框架：整体架构包括SG-Ego注释集的构建、GLEN模型的设计与训练，以及活动驱动的图编辑预测（A-GEF）任务的定义。主要模块包括场景图生成、图序列推理和动态变化建模。

关键创新：最重要的创新在于引入了时空场景图作为明确的表示方式，并提出了A-GEF问题，使得场景动态可以通过结构化转换进行推理。这与现有方法的隐式表示形成鲜明对比。

关键设计：在模型设计中，GLEN采用了图神经网络结构，结合了时序信息的处理。损失函数设计上，考虑了图序列的对齐和动态变化的预测，确保模型能够有效学习场景的演变过程。

🖼️ 关键图片

fig_0
fig_1
fig_2

📊 实验亮点

GLEN在多个基准测试中表现出色，相较于原始视频基线，推理能力提升显著。在长时间推理任务EXPLORE-Bench和新提出的A-GEF任务中，GLEN的表现优于传统的多模态语言模型，展示了其在场景动态理解中的优势。

🎯 应用场景

该研究的潜在应用领域包括智能监控、机器人导航、虚拟现实等。通过对人类活动的深入理解，能够提升这些系统的交互能力和智能水平，促进具身人工智能的发展。未来，该方法可能在更广泛的场景理解任务中发挥重要作用。

📄 摘要（原文）

Understanding human behavior while interacting with the surrounding world is crucial for many applications of embodied AI. First-person videos are particularly informative for this problem, as they well capture how activities reshape the scene over time. However, existing approaches often rely on implicit visual or language-aligned representations, disregarding structured reasoning over the scene dynamic. We argue that explicit, compositional and editable representations of human-environment interactions can play a crucial role for rich grounded activity understanding. To this end, we introduce SG-Ego, a large scale annotation set extending Ego4D with spatio-temporal scene graphs, where relations triplets are consolidated over time into explicit time-evolving descriptions of the scene state. To reason over this representation, we propose GLEN, a graph-based model that operates over scene graph sequences to both align them with textual actions and model their temporal evolution. In addition, we formulate the activity-driven graph-edit forecasting (A-GEF) problem, a novel task that casts scene dynamics as a sequence of structured transformations conditioned on ongoing actions, enabling explicit reasoning about how scenes change over time. We validate our approach across multiple downstream tasks, spanning retrieval benchmarks as EgoMCQ and EgoCVR, as well as long-horizon reasoning benchmarks as EXPLORE-Bench and the newly introduced A-GEF. GLEN achieves strong results compared to raw video baselines and it excels in reasoning settings, typically addressed only with MLLMs, while enabling controllable and structured predictions of scene dynamics driven by human activities. We believe our results establish spatio-temporal scene graphs, together with models that reason over them, as strong compositional and interpretable representations for video understanding and potentially beyond.