Evaluating Multimodal LLMs as Generalist Vision-Language-Action Agents for Drone Control: Commanding, Approaching, Tracking and Searching

📄 arXiv: 2609.01404v1 📥 PDF

作者: Jaewoo Park, Minyoung Lee, Sukmin Seo, Moonbin Yim, Hyunwook Yoon, Dohoon Ryu, Daehee Kim, Myungseo Song, Jihyuk Byun, Seunggyu Chang, Taeho Kil, Jiseob Kim, Bado Lee, Geewook Kim

分类: cs.RO, cs.AI

发布日期: 2026-09-01

备注: Preprint

💡 一句话要点

提出DroneCATS-Agent以解决无人机控制中的多模态决策问题

🎯 匹配领域: 支柱九：具身大模型 (Embodied Foundation Models)

关键词: 多模态大型语言模型 无人机控制 自主决策 目标跟踪 环境搜索 多无人机编队 边缘计算 模型评估

📋 核心要点

  1. 现有方法在无人机控制中往往限制了模型的决策能力，导致无法充分利用多模态模型的潜力。
  2. 论文提出DroneCATS-Agent架构，使MLLM成为可替换组件，允许模型在无人机控制中自主决策而无需微调。
  3. 实验表明，小型开放模型在导航上表现优于前沿模型，但在声明到达时存在失误，揭示了模型在行动协议上的不足。

📝 摘要（中文）

多模态大型语言模型（MLLMs）在图像和视频感知方面表现出色。本文探讨了将MLLM直接应用于无人机控制的能力，提出了DroneCATS-Agent架构，允许模型在不进行微调的情况下自主决策。通过评估模型在接近目标、跟踪移动物体、搜索视野外目标及指挥多无人机编队等四个核心能力的表现，发现小型开放模型在导航上表现更为可靠，但在声明到达时存在问题。DroneCATS基准旨在测量模型在边缘计算条件下的表现差距。

🔬 方法详解

问题定义：本文旨在解决无人机控制中多模态决策的不足，现有方法往往限制模型的决策空间，导致性能下降。

核心思路：提出DroneCATS-Agent架构，允许MLLM作为可替换组件，模型通过提示直接参与无人机控制决策，增强其自主性和灵活性。

技术框架：整体架构包括四个核心模块：目标接近、移动目标跟踪、视野外搜索和多无人机指挥，模型在这些模块中进行决策而无需微调。

关键创新：最重要的创新在于将MLLM直接嵌入无人机控制循环中，允许模型在不依赖于传统函数调用的情况下进行自主决策。

关键设计：模型参数设置为2B，采用特定的损失函数和网络结构，确保在边缘计算条件下的高效性和可靠性。通过自我声明到达，模型能够在复杂环境中做出更好的决策。

🖼️ 关键图片

fig_0
fig_1
fig_2

📊 实验亮点

实验结果显示，小型开放模型在接近目标时的成功率高于前沿模型，但在声明到达时出现失误，导致任务失败。通过DroneCATS基准，揭示了模型在行动协议上的不足，强调了在边缘计算条件下的性能差距。

🎯 应用场景

该研究的潜在应用领域包括无人机自主飞行、搜索与救援任务、环境监测等。通过提升无人机的自主决策能力，可以显著提高其在复杂环境中的操作效率和安全性，未来可能推动无人机技术的广泛应用。

📄 摘要（原文）

Multimodal Large Language Models (MLLMs) are strong perceivers of images and video. We ask how far that reach extends into acting: dropping an MLLM directly into a drone's control loop, with its entire action space declared solely in the prompt. Recent systems approach this setting but increasingly narrow the model's decision-making. We widen it back. We introduce DroneCATS-Agent, an architecture where the MLLM is a swappable component, and DroneCATS, a benchmark treating the model as the independent variable. Beyond merely flying toward a pixel, our agent entrusts the model to yaw and search, deliberate when unsure, and self-declare arrival---all without fine-tuning or function-calling schemas. Evaluating frontier and open models across four core capabilities---approaching a visible target, tracking a moving one, searching outside the initial view, and commanding a multi-drone fleet---reveals that even the simplest embodied settings are far from solved. Crucially, to identify what breaks first at the edge, our roster scales down to 2B parameters. The findings expose a stark paradox: it is not the flying that fails. Small open models often navigate into the success radius more reliably than frontier models, yet lose the episode by declaring arrival prematurely or not at all. Multi-drone commanding amplifies this divide, with small models failing by blindly copying a single coordinate across distinct views. Viewed as vision-language-action agents, the models' spatial perception holds up, but their action protocol does not. What separates a deployable edge model from a frontier model is not navigation, but the discipline to sustain a declared protocol and emit the correct terminating action. The open problem is closing this gap at onboard compute costs---yielding a fast model that plans persistently and knows exactly when it is done---and DroneCATS is built to measure that distance.