VisualClaw: A Real-Time, Personalized Agent for the Physical World
作者: Haoqin Tu, Jianwen Chen, Zijun Wang, Siwei Han, Juncheng Wu, Hardy Chen, Haonian Ji, Kaiwen Xiong, Jiaqi Liu, Peng Xia, Jieru Mei, Hongliang Fei, Jason Eshraghian, Zeyu Zheng, Yuyin Zhou, Huaxiu Yao, Cihang Xie
分类: cs.CV, cs.CL
发布日期: 2026-06-15
备注: H. T. and J. C. contribute to this project equally
💡 一句话要点
提出VisualClaw以解决多模态代理在视频处理中的高延迟与静态性问题
🎯 匹配领域: 支柱九：具身大模型 (Embodied Foundation Models)
关键词: 多模态代理 视觉语言模型 视频问答 自我演化 边缘计算 技能演化 动态更新
📋 核心要点
- 现有视觉语言模型在处理密集视频帧和长提示时，通常面临高延迟和高成本的问题。
- VisualClaw通过混合编码和技能演化，优化了视频处理流程，降低了部署成本并提升了代理的适应性。
- 在四个视频问答基准测试中，VisualClaw的API调用成本平均降低了98%，并在准确性上实现了显著提升。
📝 摘要（中文）
视觉语言模型（VLMs）作为复杂多模态任务的通用接口，面临高延迟、高成本、代理静态性及标准视频问答基准测试不足等挑战。本文提出VisualClaw，一个自我演化的多模态代理，采用混合编码和技能演化两大原则，显著降低了API调用成本并提升了准确性。通过在四个视频问答基准上进行测试，VisualClaw在API调用成本上平均降低了98%，并在多个设置中提高了准确性，展示了其在边缘应用中的潜力。
🔬 方法详解
问题定义：本文旨在解决现有视觉语言模型在视频处理中的高延迟和高成本问题，同时应对代理在部署后静态性和视频问答基准测试不足的挑战。
核心思路：VisualClaw的核心思路是通过混合编码技术过滤不必要的视频帧，并利用技能演化机制让代理从失败中学习，从而不断优化其技能库。
技术框架：VisualClaw的整体架构包括两个主要模块：混合编码模块和技能演化模块。混合编码模块通过级联门控过滤低信息量的帧，而技能演化模块则通过检索记忆来更新技能库。
关键创新：VisualClaw的关键创新在于其自我演化能力，允许代理在使用过程中不断学习和适应，显著提高了其在动态环境中的表现。与传统方法相比，这种设计使得代理能够更有效地利用视觉证据。
关键设计：在技术细节上，VisualClaw采用了热/冷top-k注入技术来压缩文本技能库，并通过直接拼接上下文或引导证据来条件化演化器，从而实现技能库的动态更新。
🖼️ 关键图片
📊 实验亮点
在实验中，VisualClaw在四个视频问答基准上表现出色，API调用成本平均降低了98%，相较于离线均匀8帧基线降低了25.9%。在准确性方面，使用Gemini 3 Flash时，EgoSchema的平均提升为3.85%，峰值提升达到15.80%。
🎯 应用场景
VisualClaw的研究成果在边缘计算、智能助手和机器人等领域具有广泛的应用潜力。其自我演化的能力使其能够在动态环境中提供个性化服务，提升用户体验，并在实时视频分析和交互式任务中展现出更高的效率和准确性。
📄 摘要（原文）
Vision language models are serving as general-purpose interfaces for complex multimodal tasks. However, deployment still faces three gaps: VLMs typically incur high latency and cost when processing dense video frames and long prompts, the agent scaffold remains static after deployment, and standard video-QA benchmarks do not test whether agents can use visual evidence inside tool-using workspaces. We present VisualClaw, a self-evolving multimodal agent built around two principles. First, hybrid encoding reduces deployment cost by filtering less informative streaming frames with a cascaded gate and compressing the text skill bank through hot/cold top-k injection. Second, skill evolution lets the agent learn from failures: retrieved memories condition an evolver as direct concatenated context or as guided evidence, producing skill-bank updates that help future questions. Across 4 video-QA benchmarks with 2 VLMs, VisualClaw cuts per-question API cost by an average -98% versus full-frame upload and by -25.9% over the offline uniform 8 frame baseline, while boosting accuracy in most settings, e.g., an average +3.85% and a peak +15.80% on EgoSchema with Gemini 3 Flash. To address the gap, we curate VisualClawArena, a 200-scenario multimodal agentic benchmark built through a strict five-stage pipeline; models must use video evidence, documents, dynamic updates, and executable checks inside a workspace. On VisualClawArena, the same framework with computer-use agent backends improves macro accuracy by +2.9% for Codex (GPT-5.5) and +3.2% for Claude Code (Sonnet 4.6) over no-evolution baselines, with a -9.5% cost reduction compared to the uniform-sampled baseline. These properties make VisualClaw a natural fit for edge applications, where the cascade reduces a 1-hour streaming session from ~3,600 API uploads down to only 5-20 calls and the self-evolution makes it a perfect personalized assistant.