When One Modality Is Not Enough: Multimodal Sex and Life-Stage Classification of Red Deer from Aerial RGB-Thermal Video
作者: Hugo Markoff, Christoph Praschl, Ivan Ludoški, Sara Beery, Michael Ørsted, David C. Schedl
分类: cs.CV
发布日期: 2026-08-07
备注: Accepted at the ECCV 2026 Workshop on Computer Vision for Ecology (CV4Ecology), archival proceedings track. 17 pages, 7 figures, 5 tables
💡 一句话要点
提出多模态融合方法以提高红鹿性别与生活阶段分类精度
🎯 匹配领域: 支柱九：具身大模型 (Embodied Foundation Models)
关键词: 多模态融合 红鹿分类 无人机监测 生态管理 自监督学习
📋 核心要点
- 现有的单一传感器方法在不同环境条件下表现不佳，难以准确分类红鹿的性别和生活阶段。
- 本文提出了一种多模态融合方法，通过结合RGB和热成像数据，利用自监督学习提高分类精度。
- 实验结果表明，融合方法在性别分类上优于单一传感器，分类准确率达到96.0%，显著提升了调查的有效性。
📝 摘要（中文）
随着无人机调查在野生动物种群估算中的应用日益增多，单纯的数量统计已无法满足需求，种群动态的理解需要考虑物种组成、性别比例和年龄结构等因素。本文以红鹿（Cervus elaphus）为研究对象，提出了一种融合RGB和热成像视频的多模态分类方法。该方法通过自监督的DINOv3特征融合两种传感器的数据，在每个阶段进行动物跟踪，仅在两种相机一致时确认动物身份，并通过投票机制进行物种和性别的分类。实验结果显示，融合方法在四次飞行中成功分类26只动物中的25只，性别分类准确率达到96.0%。
🔬 方法详解
问题定义：本文旨在解决在无人机调查中，单一传感器在不同环境条件下对红鹿性别和生活阶段分类的不足。现有方法在颜色和热成像下分别存在识别困难，导致分类准确率低下。
核心思路：论文提出通过融合RGB和热成像数据，利用自监督的DINOv3特征进行多模态分类。该设计旨在克服单一模态的局限性，提高分类的鲁棒性和准确性。
技术框架：整体架构包括数据采集、特征提取、动物跟踪和分类四个主要模块。首先，通过无人机采集RGB和热成像视频，然后提取自监督特征，接着进行动物跟踪，最后通过投票机制进行物种和性别分类。
关键创新：最重要的技术创新在于自监督特征融合的使用，使得在不同环境下的分类性能显著提高。与现有方法相比，该方法在处理遮挡和环境干扰时表现更为出色。
关键设计：在参数设置上，采用了DINOv3特征提取网络，损失函数设计为结合分类和回归任务的复合损失，以确保分类的准确性和鲁棒性。
🖼️ 关键图片
📊 实验亮点
实验结果显示，融合RGB和热成像数据的分类方法在四次飞行中成功分类25只动物，性别分类准确率达到96.0%。相比单一传感器，融合方法在分类性能上有显著提升，尤其在复杂环境下表现更为稳定。
🎯 应用场景
该研究的潜在应用领域包括野生动物监测、生态研究和资源管理等。通过自动化的种群结构分类，管理者可以更频繁地获取性别比例和年龄结构信息，从而做出更为科学的管理决策，提升野生动物保护的效率和效果。
📄 摘要（原文）
Aerial drone surveys increasingly support wildlife population estimation, yet a useful census is more than a count: population dynamics are defined by species composition, sex ratios and age structure, that is, by which species are present and how a herd splits into adult males, adult females and juveniles. We use red deer ($\textit{Cervus elaphus}$) as a test case, because managers act on these dynamics and because the visible cue defining adult males, the antlers, is seasonally variable. Surveys are flown nadir, high enough not to disturb the animals, so each deer occupies only a small, low-resolution patch. The two recording modalities fail in opposite conditions: in color a deer under canopy blends into the ground, while in thermal it becomes a bright blob that loses fine detail. Rather than trust either modality alone, we fuse them at every stage using self-supervised DINOv3 features. Our pipeline tracks animals in both modalities, treats an animal as confirmed only when the two cameras agree, keeps only the clear, non-occluded frames, and assigns species and sex by a vote across them; life stage is read separately from geo-referenced body size, since at survey resolution a juvenile often only differs from an adult female in size. Across four flights spanning the antler season the fused pipeline correctly classifies 25 of the 26 detected individuals (7 of 8 adult males, all 16 adult females and 2 juveniles), against 20 of 26 for either sensor alone. Multimodal species classification reaches 96.0%, while for sex classification fusing the two sensors matters most: the combined RGB+thermal model is the most robust across environments and seasons. Automating the demographic classification turns a drone flight from a count into a repeatable reading of herd structure, so the sex ratios and age structure that managers already act on can be gathered as often as a survey can be flown.