LeVJEPA: Efficient & Scalable Video Pretraining without the Heuristics

📄 arXiv: 2608.27395v1 📥 PDF

作者: Lukas Kuhn, Lucas Maes, Giuseppe Serra, Quentin Le Lidec, Yann LeCun, Randall Balestriero, Florian Buettner

分类: cs.CV, cs.AI

发布日期: 2026-08-27

💡 一句话要点

提出LeVJEPA以解决视频预训练计算成本高的问题

🎯 匹配领域: 支柱二：RL算法与架构 (RL & Architecture)

关键词: 视频预训练 自监督学习 表示学习 计算效率 深度学习

📋 核心要点

  1. 现有视频表示学习方法计算成本高，且常依赖复杂的架构设计以防止表示崩溃。
  2. LeVJEPA通过单一编码器和不变性损失，简化了视频预训练过程，消除了对复杂机制的需求。
  3. 实验结果显示，LeVJEPA在计算效率上显著提升，同时在下游任务中表现优于现有基线，尤其在运动相关任务上表现突出。

📝 摘要（中文）

视频承载了物理世界的时间结构，但从中学习表示仍然计算成本高昂。现有的自监督方法通过架构不对称性或重建掩蔽内容来防止表示崩溃。本文提出LeVJEPA，这是第一个在LeJEPA的无崩溃目标下训练的视频编码器，省略了这两种方法。通过对剪辑的全局和局部视图进行不变性损失的训练，LeVJEPA在计算成本上显著降低，同时在下游任务中表现优异。实验表明，LeVJEPA在相同数据上以5.6到20.8倍更少的预训练计算量匹配或超越了V-JEPA 2，并在运动中心基准上保持竞争力。

🔬 方法详解

问题定义：本文旨在解决现有视频预训练方法计算成本高和表示崩溃的问题。现有方法通常依赖复杂的架构和机制，导致效率低下。

核心思路：LeVJEPA通过引入不变性损失，利用单一编码器训练，避免了对复杂架构的依赖，从而实现了高效的表示学习。

技术框架：LeVJEPA的整体架构包括一个编码器和一个投影器，训练过程中通过对全局和局部视图的观察进行不变性损失的优化。

关键创新：LeVJEPA的核心创新在于其无崩溃目标的设计，允许编码器在没有不对称性的情况下进行训练，显著降低了计算成本。

关键设计：该方法的关键设计包括使用SIGReg作为正则化手段，确保表示不崩溃，以及通过均匀随机丢弃令编码器观察的token数量较小，从而提高下游任务的准确性。

🖼️ 关键图片

fig_0
fig_1
fig_2

📊 实验亮点

在相同的数据和训练周期下，LeVJEPA在ViT-S/B/L上以5.6到20.8倍更少的预训练计算量匹配或超越了V-JEPA 2，并在总FLOPs相同的情况下，在ImageNet-1K上超越了最强视频基线7.6个百分点，同时在运动中心基准上保持竞争力。

🎯 应用场景

LeVJEPA的研究成果在视频理解、行为识别和多模态学习等领域具有广泛的应用潜力。其高效的预训练方式能够降低计算资源的需求，使得视频数据在视觉预训练中的应用更加可行和高效，推动相关技术的发展。

📄 摘要（原文）

Video carries the temporal structure of the physical world, yet learning representations from it has remained computationally expensive: prevailing self-supervised methods either prevent representation collapse through architectural asymmetries, coupling an exponential-moving-average target encoder, a stop-gradient, and a capacity-limited predictor, or circumvent it by reconstructing masked content in pixel space. We introduce LeVJEPA, the first video encoder trained under LeJEPA's collapse-free objective, which dispenses with both. A single encoder is trained with an invariance loss over global and local views of a clip, regularized by SIGReg, which excludes collapse with a provable guarantee. The architecture reduces to an encoder and a projector, and the objective to a single hyperparameter. This formulation admits two properties. First, the cost of pretraining is governed by the number of tokens the encoder observes; uniform random token dropping renders this number small while simultaneously improving downstream accuracy. At matched epochs on identical data, LeVJEPA matches or surpasses V-JEPA 2 across ViT-S/B/L at 5.6 to 20.8x less pretraining compute, and at matched total FLOPs it exceeds the strongest video baseline by 7.6 points on ImageNet-1K while remaining competitive on motion-centric benchmarks. Second, since no asymmetry between branches is required, the encoder can be trained with block-causal attention at no measurable accuracy cost: temporal ordering becomes a property of the encoder itself. Against a compute-matched DINOv2 trained on frames of the same videos, LeVJEPA approaches the image-pretrained encoder on appearance-centric evaluation while nearly doubling its motion-centric accuracy. These results indicate that, once its computational overhead is removed, video becomes a viable and in several respects preferable substrate for general-purpose visual pretraining.