A Reverse Sign Language Dictionary: Open-Vocabulary Sign Recognition from Continuous Signing via Video Captioning and Description Retrieval

📄 arXiv: 2609.03788v1 📥 PDF

作者: Santiago Poveda-Gutiérrez, Hideki Nakayama, Mayumi Bono

分类: cs.CV, cs.CL

发布日期: 2026-09-03

备注: 4 pages, 2 figures, 1 table. Extended version of an abstract presented at the BU-SHI workshop (Broadening the Users: A Cross-Disciplinary Roadmap for Social Humanoid Interaction), IEEE RO-MAN 2026, Kitakyushu, Japan, 28 August 2026. The workshop is non-archival; no proceedings

💡 一句话要点

提出一种反向手语字典以解决开放词汇手语识别问题

🎯 匹配领域: 支柱三：空间感知与语义 (Perception & Semantics)

关键词: 手语识别 开放词汇 多模态学习 视觉-语言模型 描述生成 机器学习 自然语言处理

📋 核心要点

  1. 现有的孤立手语识别方法无法处理训练中未见的手势，限制了其应用的灵活性和广泛性。
  2. 论文提出通过生成手势的逐步描述并检索目标描述，构建反向手语字典，实现开放词汇的手语识别。
  3. 实验结果显示，经过微调后，已见类别的检索准确率从4.5%提升至49%，未见类别的检索准确率也显著提高。

📝 摘要（中文）

孤立手语识别（ISLR）通常被视为对标签的封闭集分类，无法推广到训练中未见的手势，并将每次部署绑定到带有标签的词汇表上。本文提出了一种新方法，通过使用开放权重的视觉-语言模型对手势进行逐步描述的字幕生成，并利用多语言句子编码器从目标描述词汇中检索最接近的条目，从而实现从连续手语中提取手势的识别。实验表明，经过微调的字幕生成器在已见类别的检索上显著提高，未见类别的检索也有显著提升，这是封闭集分类器无法参与的领域。该研究是首次在没有标签监督的情况下实现基于描述的开放词汇手语查找。

🔬 方法详解

问题定义：本文旨在解决孤立手语识别中的封闭集分类问题，现有方法无法处理训练中未见的手势，限制了手语识别的灵活性和适用性。

核心思路：通过生成手势的逐步描述并利用多语言句子编码器检索目标描述，构建一个无需标签监督的反向手语字典，从而实现开放词汇的手语识别。

技术框架：整体架构包括两个主要模块：首先，使用开放权重的视觉-语言模型对手势进行字幕生成；其次，利用多语言句子编码器从目标描述词汇中检索最接近的条目。

关键创新：本研究的最大创新在于首次实现了在没有标签监督的情况下，从连续手语中进行基于描述的开放词汇手语查找，这与传统的封闭集分类方法有本质区别。

关键设计：在实验中，采用了微调的字幕生成器，优化了语言和视觉塔的参数设置，显著提高了检索性能，尤其是在未见类别的检索上。

🖼️ 关键图片

fig_0
img_1
img_2

📊 实验亮点

实验结果显示，经过微调的字幕生成器在已见类别的检索准确率从4.5%提升至49%，在未见类别的检索准确率也从11.5%提升至21.0%，显著优于未训练的基线，且在某些测试集上与标准的封闭集分类器表现相当。

🎯 应用场景

该研究具有广泛的应用潜力，尤其在手语翻译、教育和人机交互等领域。通过实现开放词汇的手语识别，能够更好地服务于手语使用者，提升其与技术的互动体验，未来可能对手语学习和交流方式产生深远影响。

📄 摘要（原文）

Isolated Sign Language Recognition (ISLR) is conventionally cast as closed-set classification over gloss labels, which cannot generalize to signs unseen in training and ties every deployment to a gloss-annotated lexicon. We instead recognize signs extracted from continuous signing by (1) captioning a sign-level clip into a free-form procedural description of the articulation with an open-weight vision-language model, and (2) retrieving the closest entry from a vocabulary of target descriptions with a multilingual sentence encoder: a reverse sign language dictionary that needs no gloss supervision and admits an open vocabulary. On 1,300 sign-level segments from a Japanese Sign Language (JSL) dialogue corpus annotated with procedural descriptions (against a 2% top-10 chance floor over the 503-entry target vocabulary), fine-tuning the captioner substantially improves seen-class retrieval: language and vision tower fine-tuning raises top-10 retrieval on seen classes from 4.5% (untrained) to 49%, becoming statistically indistinguishable from a standard supervised closed-set classifier (I3D) on two of the three test sets where a closed-set classifier can be evaluated at all. More importantly, unseen-class retrieval also improves significantly over the untrained pipeline (11.5% -> 21.0% top-10, p=0.0094), a regime in which the closed-set classifier cannot participate. A matcher-side empirical upper-bound analysis shows the sentence encoder already recovers close to 100% of paraphrased gold descriptions, locating a gap in captioning quality that we aim to address in future work. To our knowledge this is the first description-based, open-vocabulary sign lookup from continuous signing without gloss supervision, and the first for JSL.