WavLLM: Towards Robust and Adaptive Speech Large Language Model

📄 arXiv: 2404.00656v3 📥 PDF

作者: Shujie Hu, Long Zhou, Shujie Liu, Sanyuan Chen, Lingwei Meng, Hongkun Hao, Jing Pan, Xunying Liu, Jinyu Li, Sunit Sivasankaran, Linquan Liu, Furu Wei

分类: cs.CL, cs.AI, cs.SD, eess.AS

发布日期: 2024-03-31 (更新: 2024-09-21)

备注: accepted by EMNLP2024 findings

💡 一句话要点

提出WavLLM以解决语音任务中的泛化与复杂性问题

🎯 匹配领域: 支柱二：RL算法与架构 (RL & Architecture) 支柱九：具身大模型 (Embodied Foundation Models)

关键词: 语音识别 多模态学习 课程学习 双编码器 LoRA适配器 泛化能力 复杂任务

📋 核心要点

  1. 现有的语音处理模型在复杂任务和多样化上下文中的泛化能力不足，难以有效执行复杂的听觉任务。
  2. WavLLM通过双编码器架构解耦语音信息，并引入基于提示的LoRA适配器，采用两阶段课程学习优化模型性能。
  3. 实验结果显示，WavLLM在多项语音任务上达到了最先进的性能，尤其在高考英语听力理解任务中表现出色。

📝 摘要（中文）

近年来，大型语言模型（LLMs）的进展彻底改变了自然语言处理领域，逐步扩展到多模态感知和生成。然而，将听觉能力有效整合到LLMs中面临重大挑战，尤其是在不同上下文中的泛化能力和复杂听觉任务的执行方面。本文提出了WavLLM，一个具有双编码器和基于提示的LoRA权重适配器的鲁棒自适应语音大型语言模型，采用两阶段课程学习方法进行优化。通过双编码器，WavLLM解耦不同类型的语音信息，利用Whisper编码器处理语音的语义内容，WavLM编码器捕捉说话者身份的独特特征。实验结果表明，该模型在多项语音任务上实现了最先进的性能，展现出强大的泛化能力。

🔬 方法详解

问题定义：本文旨在解决现有大型语言模型在语音任务中泛化能力不足和复杂任务执行困难的问题。现有方法在多样化上下文中表现不佳，难以处理复杂的听觉任务。

核心思路：WavLLM的核心思路是通过双编码器架构解耦语音信息，分别处理语音的语义内容和说话者身份特征，同时引入基于提示的LoRA适配器，以增强模型的灵活性和任务适应性。

技术框架：WavLLM的整体架构包括两个主要编码器：Whisper编码器用于语义内容处理，WavLM编码器用于说话者特征捕捉。模型采用两阶段课程学习，首先在基础单任务上优化，然后在复杂多任务上进行训练。

关键创新：WavLLM的主要创新在于双编码器的设计和基于提示的LoRA适配器的引入，这使得模型能够更好地处理复杂的语音任务，并在不同任务之间灵活切换。

关键设计：模型的关键设计包括课程学习的两阶段策略，损失函数的优化，以及在多任务训练阶段的参数设置，确保模型在不同任务上均能达到优异的性能。

🖼️ 关键图片

fig_0
fig_1
fig_2

📊 实验亮点

实验结果表明，WavLLM在多个语音任务上实现了最先进的性能，尤其在高考英语听力理解任务中表现出色，且无需专门训练即可完成这些任务，展现出强大的泛化能力。

🎯 应用场景

WavLLM的研究成果在语音识别、语音翻译、说话人验证等多个领域具有广泛的应用潜力。其强大的泛化能力和适应性使其能够在多样化的实际场景中有效执行复杂的听觉任务，未来可能推动智能助手、教育和客服等行业的发展。

📄 摘要（原文）

The recent advancements in large language models (LLMs) have revolutionized the field of natural language processing, progressively broadening their scope to multimodal perception and generation. However, effectively integrating listening capabilities into LLMs poses significant challenges, particularly with respect to generalizing across varied contexts and executing complex auditory tasks. In this work, we introduce WavLLM, a robust and adaptive speech large language model with dual encoders, and a prompt-aware LoRA weight adapter, optimized by a two-stage curriculum learning approach. Leveraging dual encoders, we decouple different types of speech information, utilizing a Whisper encoder to process the semantic content of speech, and a WavLM encoder to capture the unique characteristics of the speaker's identity. Within the curriculum learning framework, WavLLM first builds its foundational capabilities by optimizing on mixed elementary single tasks, followed by advanced multi-task training on more complex tasks such as combinations of the elementary tasks. To enhance the flexibility and adherence to different tasks and instructions, a prompt-aware LoRA weight adapter is introduced in the second advanced multi-task training stage. We validate the proposed model on universal speech benchmarks including tasks such as ASR, ST, SV, ER, and also apply it to specialized datasets like Gaokao English listening comprehension set for SQA, and speech Chain-of-Thought (CoT) evaluation set. Experiments demonstrate that the proposed model achieves state-of-the-art performance across a range of speech tasks on the same model size, exhibiting robust generalization capabilities in executing complex tasks using CoT approach. Furthermore, our model successfully completes Gaokao tasks without specialized training. The codes, models, audio, and Gaokao evaluation set can be accessed at \url{aka.ms/wavllm}.