Select, Compress, Reinvest: A Controlled Study of Visual-Token Allocation in Long-Video MLLMs
作者: Prakhar Khatri
分类: cs.CV, cs.CL
发布日期: 2026-09-03
备注: 16 pages, 6 figures. Code and data: https://github.com/codeprakhar25/omp-keyframe-sampling
💡 一句话要点
提出视觉令牌分配方法以优化长视频MLLM性能
🎯 匹配领域: 支柱九：具身大模型 (Embodied Foundation Models)
关键词: 长视频理解 视觉令牌分配 多模态学习 帧选择 空间压缩
📋 核心要点
- 长视频模型在处理大量帧时面临选择和压缩的挑战，现有方法未能充分利用帧选择的潜力。
- 本文通过控制实验，逐步分析帧选择、空间压缩和再投资的影响，提出了一种系统化的评估框架。
- 实验结果表明，选择策略显著提升性能，压缩带来的损失极小，而再投资则能进一步提高准确性。
📝 摘要（中文）
长视频语言模型无法处理每一帧，通常只保留一小部分帧。本文研究了帧选择、空间压缩和节省令牌的再投资对模型性能的影响。通过对比六种选择规则和多个基准，发现选择策略对性能影响最大，且压缩几乎不损失精度。再投资节省的令牌可进一步提升准确性，显示出在控制环境下进行比较的重要性。
🔬 方法详解
问题定义：本文旨在解决长视频语言模型在处理大量帧时的帧选择和压缩问题。现有方法通常将帧选择视为预处理细节，未能充分评估其对模型性能的影响。
核心思路：论文通过控制实验逐一分析帧选择、空间压缩和再投资的影响，探讨如何在固定预算下优化模型性能。选择策略被认为是影响性能的最大因素。
技术框架：研究采用了六种训练无关的选择规则，结合三个长视频基准和两个回答模型，系统评估不同策略的效果。整体流程包括选择、压缩和再投资三个主要阶段。
关键创新：最重要的创新在于通过控制实验逐一分析不同决策对模型性能的影响，尤其是选择策略的显著性。与现有方法相比，本文提供了更为细致的比较和分析。
关键设计：在实验中，采用了固定的帧评分、提示边界和分辨率策略，确保每次只改变一个变量。选择策略的有效性通过与传统稀疏近似算法的比较得以验证。
🖼️ 关键图片
📊 实验亮点
实验结果显示，在LongVideoBench的小时长视频基准上，采用八帧选择的模型比均匀间隔的十六帧提升了6.9分。此外，传统的稀疏近似算法Orthogonal Matching Pursuit在所有基准中表现出色，接近或超过了专门设计的选择器的性能。
🎯 应用场景
该研究的潜在应用领域包括视频理解、自动摘要生成和多模态学习等。通过优化长视频处理，能够提升相关应用的准确性和效率，具有重要的实际价值和未来影响。
📄 摘要（原文）
Long-video language models cannot look at every frame: an hour sampled once per second is 3,600 images, and a system keeps only a small fixed slice of that pool. Which frames survive that slice is usually treated as a preprocessing detail; we test whether it should be. Published selectors make the comparison hard because they change the frame scorer, the prompt boundary, the resolution policy, and the answering model all at once. We hold each fixed and vary one decision at a time: selection, spatial compression, and reinvestment of the savings, across six training-free selection rules, three long-video benchmarks, and two answering models. Selection is the largest single lever: on LongVideoBench's hour-long bin, eight query-selected frames beat sixteen uniformly spaced ones by 6.9 points, and Orthogonal Matching Pursuit, an unmodified decades-old sparse-approximation algorithm, matches or comes within a point of every purpose-built selector we compare it against, across all three benchmarks. Compression is close to free: halving each frame's spatial budget at fixed timestamps costs at most 0.44 points. Reinvestment is where that budget turns back into accuracy: spending the freed tokens on twice as many compressed frames, at a measured cost no higher than the original eight, returns a further two to three points; compression only pays off once its savings are spent this way. Along the way, an implementation bug in our own AKS baseline and a 0.07 to 3.74 point gap between two harnesses running the same published rules at the same budget show why these comparisons need to happen inside one controlled harness rather than across papers.