An Equivariant Pretrained Transformer for Unified 3D Molecular Representation Learning

📄 arXiv: 2402.12714v2 📥 PDF

作者: Rui Jiao, Xiangzhe Kong, Li Zhang, Ziyang Yu, Fangyuan Ren, Wenjuan Tan, Wenbing Huang, Yang Liu

分类: cs.LG, physics.chem-ph

发布日期: 2024-02-20 (更新: 2025-02-24)

💡 一句话要点

提出等变预训练变换器以解决跨领域3D分子表示学习问题

🎯 匹配领域: 支柱二：RL算法与架构 (RL & Architecture) 支柱九：具身大模型 (Embodied Foundation Models)

📋 核心要点

  1. 现有方法通常专注于特定领域的预训练，未能有效利用跨领域知识，限制了模型的通用性和性能。
  2. 提出的EPT模型基于E(3)-等变变换器，能够同时处理原子级和块级特征，采用块级去噪任务作为预训练目标。
  3. EPT在配体结合亲和力预测等任务中表现优异，超越了现有最先进方法，并成功识别出针对3CL蛋白酶的小分子药物候选。
  4. method_zh

📝 摘要（中文）

在大量未标记的3D分子上进行预训练已在多种科学应用中展现出优越性。然而，现有研究通常集中于特定领域的模型预训练，未能有效利用跨领域知识。为此，本文提出了等变预训练变换器（EPT），这是一个可以从多个领域的3D分子进行预训练的全原子基础模型。EPT基于E(3)-等变变换器，能够处理原子级信息并结合块级特征。通过构建一个包含小分子、蛋白质及其复合物的大规模数据集，EPT在配体结合亲和力预测等下游任务中显著超越了现有的最先进方法，并在药物发现中展示了其强大的能力。

🖼️ 关键图片

fig_0
fig_1
fig_2

📄 摘要（原文）

Pretraining on a large number of unlabeled 3D molecules has showcased superiority in various scientific applications. However, prior efforts typically focus on pretraining models in a specific domain, either proteins or small molecules, missing the opportunity to leverage cross-domain knowledge. To mitigate this gap, we introduce Equivariant Pretrained Transformer (EPT), an all-atom foundation model that can be pretrained from multiple domain 3D molecules. Built upon an E(3)-equivariant transformer, EPT is able to not only process atom-level information but also incorporate block-level features (e.g. residuals in proteins). Additionally, we employ a block-level denoising task, rather than the conventional atom-level denoising, as the pretraining objective. To pretrain EPT, we construct a large-scale dataset of 5.89M entries, comprising small molecules, proteins, protein-protein complexes, and protein-molecule complexes. Experimental evaluations on downstream tasks including ligand binding affinity prediction, protein property prediction, and molecular property prediction, show that EPT significantly outperforms previous state-of-the-art methods in the first task and achieves competitively superior performance for the remaining two tasks. Furthermore, we demonstrate the potential of EPT in identifying small molecule drug candidates targeting 3CL protease, a critical target in the replication of SARS-CoV-2. Among 1,978 FDA-approved drugs, EPT ranks 7 out of 8 known anti-COVID-19 drugs in the top 200, indicating the high recall of EPT. By using Molecular Dynamics (MD) simulations, EPT further discoveries 7 novel compounds whose binding affinities are higher than that of the top-ranked known anti-COVID-19 drug, showcasing its powerful capabilities in drug discovery.