Temporally Grounded Compositional Camera Motion Understanding via Geometric Knowledge Distillation

📄 arXiv: 2608.10932v1 📥 PDF

作者: Dazhao Du, Shiyan Du, Jian Liu, Yongjian Yu, Bohai Gu, Tao Han, Hualuo Liu, Eric Liu, Yujia Zhang, Xi Chen, Song Guo

分类: cs.CV, cs.AI

发布日期: 2026-08-11

🔗 代码/项目: PROJECT_PAGE

💡 一句话要点

提出CamDistill以解决视频中相机运动理解问题

🎯 匹配领域: 支柱二：RL算法与架构 (RL & Architecture) 支柱九：具身大模型 (Embodied Foundation Models)

关键词: 相机运动理解 时间上有依据 组合识别 几何知识蒸馏 视频感知 数据集构建 多模态学习

📋 核心要点

  1. 现有方法通常对视频片段进行整体标签化，无法捕捉相机运动的细粒度变化和复合运动特性。
  2. 本文提出CamChoreo数据集和CamDistill方法，通过时间上有依据的组合识别来理解相机运动，提升了模型的识别能力。
  3. 实验结果表明，CamDistill在不运行3D模型的情况下，达到了与直接特征注入相当的准确性，展示了显著的性能提升。

📝 摘要（中文）

理解相机运动是视频感知的基础，具有空间智能和可控视频生成等应用。现有方法通常对整个视频片段进行标签化，忽视了相机运动的两个特性：在镜头内可能发生变化，以及多个运动可以同时存在。为此，本文将相机运动理解形式化为时间上有依据的组合识别，要求模型能够定位运动一致的时间段，并识别每个时间段内的所有活动运动。我们引入了CamChoreo，一个包含4229个真实单镜头片段的基准数据集，并提出了CamDistill，通过蒸馏几何知识来训练轻量级相机标记，从而在推理时无需运行3D模型，显著提升了相机运动理解的准确性。

🔬 方法详解

问题定义：本文旨在解决现有视频理解方法在相机运动识别中的不足，特别是对运动变化和复合运动的忽视。

核心思路：提出将相机运动理解视为时间上有依据的组合识别，利用CamChoreo数据集进行训练，并通过CamDistill蒸馏几何知识以简化模型推理过程。

技术框架：整体架构包括数据集构建、模型训练和推理三个阶段。CamChoreo提供了丰富的标注数据，而CamDistill则在训练过程中将几何知识蒸馏到轻量级相机标记中。

关键创新：CamDistill的主要创新在于通过蒸馏几何知识来替代直接特征注入，避免了在推理阶段对3D模型的依赖，从而提高了效率。

关键设计：在训练过程中，采用了特定的损失函数来优化相机标记的表示，同时确保模型能够有效捕捉到运动的一致性和复合性。

🖼️ 关键图片

fig_0
fig_1
fig_2

📊 实验亮点

实验结果显示，CamDistill在相机运动理解任务中达到了与基线CamInject相当的准确性，且在推理时无需运行3D模型，显著降低了计算成本，提升了处理效率。

🎯 应用场景

该研究在视频理解、增强现实和机器人导航等领域具有广泛的应用潜力。通过更准确地理解相机运动，能够提升视频生成的控制能力和空间智能的表现，推动相关技术的发展。

📄 摘要（原文）

Understanding camera motion is fundamental to video perception, with applications in spatial intelligence and controllable video generation. Multimodal large language models (MLLMs) provide a natural interface for this task, but existing work typically assigns one or more labels to an entire clip. Such clip-level recognition overlooks two defining properties of real camera motion: it can change within a shot, and multiple movements can occur simultaneously. We therefore formulate camera-motion understanding as temporally grounded, compositional recognition, which requires a model to localize motion-consistent intervals and identify every movement active within each interval. We introduce CamChoreo, a benchmark of 4,229 real single-shot clips with expert-annotated temporal segments. Its annotations use a compact vocabulary of 20 direction-aware labels, and nearly half of the segments contain compound camera motion, with multiple movement primitives active simultaneously. Recognizing such fine-grained, compositional motion is hard for current MLLMs, whose visual encoders emphasize semantic content rather than the geometric evidence on which camera motion depends. Directly injecting features from a frozen 3D foundation model addresses this gap, but requires running the expensive geometry model on every input; we refer to this baseline as CamInject. We instead propose CamDistill, which distills the same geometric knowledge into lightweight camera tokens during training and removes the 3D model at inference. CamDistill matches the accuracy of direct feature injection without running the 3D teacher at inference. Together, CamChoreo and CamDistill advance camera-motion understanding from clip-level labeling to temporally grounded, compositional recognition. Project page: https://ddz16.github.io/cammotion.github.io/.