Learning to Evolve Scenes: Reasoning about Human Activities with Scene Graphs

📄 arXiv: 2607.02425 📥 PDF

作者: Francesca Pistilli, Simone Alberto Peirone, Giuseppe Averta

分类: cs.CV

发布日期: 2026-07-05

💡 一句话要点

提出SG-Ego和GLEN以解决人类活动理解问题

🎯 匹配领域: 支柱六：视频提取与匹配 (Video Extraction) 支柱九：具身大模型 (Embodied Foundation Models)

关键词: 时空场景图 人类活动理解 图神经网络 结构化推理 视频理解 具身AI 活动驱动预测

📋 核心要点

  1. 现有方法多依赖隐式表示，缺乏对场景动态的结构化推理，限制了人类活动理解的深度。
  2. 提出SG-Ego和GLEN，利用时空场景图进行明确的活动理解，并引入A-GEF任务以推理场景变化。
  3. GLEN在多个基准测试中表现优异，尤其在长时间推理任务中，相比于原始视频基线有显著提升。

📝 摘要（中文）

理解人类在与周围世界互动时的行为对于许多具身AI应用至关重要。第一人称视频在捕捉活动如何随时间重塑场景方面特别有效。然而，现有方法往往依赖于隐式的视觉或语言对齐表示，忽视了对场景动态的结构化推理。本文提出SG-Ego，一个扩展Ego4D的大规模注释集，利用时空场景图明确描述场景状态的演变。我们还提出GLEN，一个基于图的模型，能够对场景图序列进行推理，并与文本动作对齐。通过活动驱动的图编辑预测（A-GEF）问题，我们能够明确推理场景如何随时间变化。实验结果表明，GLEN在多个下游任务中表现优异，尤其是在推理设置中超越了传统的多模态语言模型。

🔬 方法详解

问题定义：本文旨在解决人类活动理解中的结构化推理问题。现有方法往往依赖隐式的视觉或语言表示，无法有效捕捉场景动态的演变。

核心思路：通过引入SG-Ego和GLEN，利用时空场景图明确表示人类与环境的互动，进而实现对场景动态的结构化推理。设计上强调可编辑性和可组合性，以便更好地理解活动的演变。

技术框架：整体架构包括SG-Ego注释集的构建、GLEN模型的设计和活动驱动的图编辑预测（A-GEF）任务。SG-Ego提供了时空场景图，GLEN则在此基础上进行推理和预测。

关键创新：最重要的创新在于引入了时空场景图作为明确的表示形式，并提出了A-GEF任务，使得场景动态的推理变得结构化和可控。这与现有方法的隐式表示形成鲜明对比。

关键设计：在模型设计中，GLEN采用图神经网络结构，结合损失函数以优化场景图与文本动作的对齐，同时考虑时间演变的特性。

🖼️ 关键图片

fig_0
fig_1
fig_2

📊 实验亮点

GLEN在多个下游任务中表现出色，尤其在EgoMCQ和EgoCVR等检索基准上超越了原始视频基线，提升幅度显著。此外，在长时间推理基准EXPLORE-Bench和新引入的A-GEF任务中，GLEN的推理能力也表现优异，显示出其在复杂场景理解中的优势。

🎯 应用场景

该研究在具身AI、视频理解和人机交互等领域具有广泛的应用潜力。通过提供明确的场景动态表示，能够提升机器人在复杂环境中的决策能力，并为智能监控、虚拟现实等应用提供更深层次的理解和交互能力。

📄 摘要（原文）

Understanding human behavior while interacting with the surrounding world is crucial for many applications of embodied AI. First-person videos are particularly informative for this problem, as they well capture how activities reshape the scene over time. However, existing approaches often rely on implicit visual or language-aligned representations, disregarding structured reasoning over the scene dynamic. We argue that explicit, compositional and editable representations of human-environment interactions can play a crucial role for rich grounded activity understanding. To this end, we introduce SG-Ego, a large scale annotation set extending Ego4D with spatio-temporal scene graphs, where relations triplets are consolidated over time into explicit time-evolving descriptions of the scene state. To reason over this representation, we propose GLEN, a graph-based model that operates over scene graph sequences to both align them with textual actions and model their temporal evolution. In addition, we formulate the activity-driven graph-edit forecasting (A-GEF) problem, a novel task that casts scene dynamics as a sequence of structured transformations conditioned on ongoing actions, enabling explicit reasoning about how scenes change over time. We validate our approach across multiple downstream tasks, spanning retrieval benchmarks as EgoMCQ and EgoCVR, as well as long-horizon reasoning benchmarks as EXPLORE-Bench and the newly introduced A-GEF. GLEN achieves strong results compared to raw video baselines and it excels in reasoning settings, typically addressed only with MLLMs, while enabling controllable and structured predictions of scene dynamics driven by human activities. We believe our results establish spatio-temporal scene graphs, together with models that reason over them, as strong compositional and interpretable representations for video understanding and potentially beyond.