UniGraph: Learning a Unified Cross-Domain Foundation Model for Text-Attributed Graphs

📄 arXiv: 2402.13630v3 📥 PDF

作者: Yufei He, Yuan Sui, Xiaoxin He, Bryan Hooi

分类: cs.LG

发布日期: 2024-02-21 (更新: 2025-01-20)

备注: KDD 2025

💡 一句话要点

提出UniGraph以解决跨领域图学习的知识迁移问题

🎯 匹配领域: 支柱二：RL算法与架构 (RL & Architecture) 支柱九：具身大模型 (Embodied Foundation Models)

关键词: 图学习 基础模型 自监督学习 文本属性图 图神经网络 知识迁移 零样本学习

📋 核心要点

  1. 现有图学习方法主要集中于单一图模型，缺乏跨领域知识迁移能力，限制了其应用范围。
  2. 本文提出UniGraph框架，通过文本属性图（TAGs）实现节点表示的统一，采用语言模型与图神经网络的级联架构。
  3. 实验结果表明，UniGraph在自监督表示学习、少量样本迁移和零样本迁移方面表现优异，超越了传统的图神经网络。

📝 摘要（中文）

基础模型如ChatGPT和GPT-4已在人工智能领域引发革命，展现出跨任务和应用的卓越泛化能力。然而，图学习主要集中于单图模型，缺乏跨领域知识迁移能力。本文提出UniGraph框架，利用文本属性图（TAGs）作为统一媒介，设计了一种能够在不同领域和任务中泛化的基础模型。通过创新的语言模型与图神经网络的级联架构，以及针对TAGs的大规模自监督学习预训练算法，UniGraph在多种图学习任务中表现出色，超越或匹配了经过监督训练的图神经网络的性能。

🔬 方法详解

问题定义：本文旨在解决现有图学习方法在不同领域间知识迁移不足的问题。现有方法多为单图模型，无法有效利用图结构的多样性和复杂性。

核心思路：通过将文本作为统一媒介，利用文本属性图（TAGs）来实现节点表示的统一，进而设计出能够泛化到未见图和任务的基础模型。

技术框架：UniGraph框架包括语言模型（LMs）和图神经网络（GNNs）的级联架构，首先通过语言模型提取文本特征，然后将其与图结构结合，形成统一的节点表示。

关键创新：提出了针对TAGs的首个大规模自监督学习预训练算法，基于掩蔽图建模（Masked Graph Modeling），并引入图指令调优以实现零样本预测能力。

关键设计：在网络结构上，采用了级联的LM和GNN架构，设计了特定的损失函数以优化节点表示，同时在预训练阶段使用了掩蔽策略以增强模型的泛化能力。

🖼️ 关键图片

fig_0

📊 实验亮点

实验结果显示，UniGraph在多种图学习任务中表现优异，尤其是在自监督表示学习上，能够在未见图上实现超过90%的准确率，且在少量样本迁移和零样本迁移中均表现出色，超越了传统的图神经网络模型。

🎯 应用场景

该研究的潜在应用领域包括社交网络分析、生物信息学、推荐系统等，能够有效提升跨领域图数据的处理能力。未来，UniGraph有望推动图学习技术的广泛应用，尤其是在需要快速适应新任务的场景中。

📄 摘要（原文）

Foundation models like ChatGPT and GPT-4 have revolutionized artificial intelligence, exhibiting remarkable abilities to generalize across a wide array of tasks and applications beyond their initial training objectives. However, graph learning has predominantly focused on single-graph models, tailored to specific tasks or datasets, lacking the ability to transfer learned knowledge to different domains. This limitation stems from the inherent complexity and diversity of graph structures, along with the different feature and label spaces specific to graph data. In this paper, we recognize text as an effective unifying medium and employ Text-Attributed Graphs (TAGs) to leverage this potential. We present our UniGraph framework, designed to learn a foundation model for TAGs, which is capable of generalizing to unseen graphs and tasks across diverse domains. Unlike single-graph models that use pre-computed node features of varying dimensions as input, our approach leverages textual features for unifying node representations, even for graphs such as molecular graphs that do not naturally have textual features. We propose a novel cascaded architecture of Language Models (LMs) and Graph Neural Networks (GNNs) as backbone networks. Additionally, we propose the first pre-training algorithm specifically designed for large-scale self-supervised learning on TAGs, based on Masked Graph Modeling. We introduce graph instruction tuning using Large Language Models (LLMs) to enable zero-shot prediction ability. Our comprehensive experiments across various graph learning tasks and domains demonstrate the model's effectiveness in self-supervised representation learning on unseen graphs, few-shot in-context transfer, and zero-shot transfer, even surpassing or matching the performance of GNNs that have undergone supervised training on target datasets.