Towards a Unified Transformer-based Framework for Scene Graph Generation and Human-object Interaction Detection

📄 arXiv: 2311.01755v1 📥 PDF

作者: Tao He, Lianli Gao, Jingkuan Song, Yuan-Fang Li

分类: cs.CV

发布日期: 2023-11-03

💡 一句话要点

提出SG2HOI+框架以统一场景图生成与人机交互检测

🎯 匹配领域: 支柱五：交互与反应 (Interaction & Reaction)

关键词: 场景图生成 人机交互检测 Transformer 多任务学习 计算机视觉

📋 核心要点

  1. 现有方法将场景图生成和人机交互检测视为独立任务，导致模型无法共享信息，影响性能。
  2. 本文提出SG2HOI+模型，通过两个交互式Transformer统一处理SGG和HOI任务，利用视觉关系增强人机交互推理。
  3. 在多个基准数据集上，SG2HOI+表现优于现有单阶段SGG模型，并在HOI任务上与最先进方法相当，联合训练显著提升了性能。

📝 摘要（中文）

场景图生成（SGG）和人机交互（HOI）检测是两个重要的视觉任务，分别旨在定位和识别物体之间的关系以及人类与物体之间的交互。现有研究将这两个任务视为独立的任务，导致开发出针对特定数据集的模型。本文提出SG2HOI+，一个基于Transformer架构的统一模型，通过两个交互式层次Transformer无缝整合SGG和HOI检测任务。实验结果表明，SG2HOI+在多个基准数据集上表现优异，且在联合训练时相较于个体训练方法显著提升了两项任务的性能。

🔬 方法详解

问题定义：本文旨在解决场景图生成（SGG）与人机交互（HOI）检测任务之间的独立性问题。现有方法未能有效利用视觉关系，导致推理能力不足。

核心思路：提出SG2HOI+模型，通过两个交互式Transformer将SGG和HOI任务统一处理，利用生成的关系三元组来推断人机交互，增强了任务间的关联性。

技术框架：SG2HOI+模型包含两个主要模块：首先是关系Transformer，用于从视觉特征生成关系三元组；其次是解码器Transformer，根据生成的关系三元组预测人机交互。

关键创新：SG2HOI+的最大创新在于将SGG与HOI任务整合为一个统一的模型，利用视觉关系信息提升了人机交互的推理能力，与传统的任务特定模型形成鲜明对比。

关键设计：模型采用层次化的Transformer结构，设置了适当的损失函数以平衡SGG与HOI任务的训练，确保两者在联合训练中互相促进。

🖼️ 关键图片

fig_0
fig_1
fig_2

📊 实验亮点

SG2HOI+模型在Visual Genome、V-COCO和HICO-DET等基准数据集上表现优异，相较于现有单阶段SGG模型，性能提升显著。同时，在HOI任务上与最先进的方法相比，表现也相当，联合训练带来了显著的性能提升，验证了模型的有效性。

🎯 应用场景

该研究的潜在应用领域包括智能监控、机器人视觉、增强现实等，能够提升系统对复杂场景的理解能力，促进人机交互的智能化和自然化。未来，该框架可能推动多任务学习在计算机视觉领域的进一步发展，提升模型的通用性和适应性。

📄 摘要（原文）

Scene graph generation (SGG) and human-object interaction (HOI) detection are two important visual tasks aiming at localising and recognising relationships between objects, and interactions between humans and objects, respectively. Prevailing works treat these tasks as distinct tasks, leading to the development of task-specific models tailored to individual datasets. However, we posit that the presence of visual relationships can furnish crucial contextual and intricate relational cues that significantly augment the inference of human-object interactions. This motivates us to think if there is a natural intrinsic relationship between the two tasks, where scene graphs can serve as a source for inferring human-object interactions. In light of this, we introduce SG2HOI+, a unified one-step model based on the Transformer architecture. Our approach employs two interactive hierarchical Transformers to seamlessly unify the tasks of SGG and HOI detection. Concretely, we initiate a relation Transformer tasked with generating relation triples from a suite of visual features. Subsequently, we employ another transformer-based decoder to predict human-object interactions based on the generated relation triples. A comprehensive series of experiments conducted across established benchmark datasets including Visual Genome, V-COCO, and HICO-DET demonstrates the compelling performance of our SG2HOI+ model in comparison to prevalent one-stage SGG models. Remarkably, our approach achieves competitive performance when compared to state-of-the-art HOI methods. Additionally, we observe that our SG2HOI+ jointly trained on both SGG and HOI tasks in an end-to-end manner yields substantial improvements for both tasks compared to individualized training paradigms.