VoxGenesis: Unsupervised Discovery of Latent Speaker Manifold for Speech Synthesis

📄 arXiv: 2403.00529v1 📥 PDF

作者: Weiwei Lin, Chenhang He, Man-Wai Mak, Jiachen Lian, Kong Aik Lee

分类: cs.SD, cs.LG, eess.AS

发布日期: 2024-03-01

备注: preprint

💡 一句话要点

提出VoxGenesis以解决无监督语音合成问题

🎯 匹配领域: 支柱一：机器人控制 (Robot Control)

关键词: 无监督学习 语音合成 潜在流形 声音编辑 高斯分布 深度学习 特征提取

📋 核心要点

  1. 现有的语音合成方法依赖于有监督的说话人建模，难以获取准确的情感、语调和说话风格标签。
  2. VoxGenesis通过无监督学习发现潜在的说话人流形，利用高斯分布生成不同特征的说话人。
  3. 实验表明，VoxGenesis在多样性和真实感上显著优于以往方法，且潜在空间操控不影响语音质量。

📝 摘要（中文）

实现对人类声音的细腻和准确模拟一直是人工智能的长期目标。尽管近年来取得了显著进展，但主流的语音合成模型仍依赖于有监督的说话人建模和显式的参考语句。针对这一问题，本文提出了VoxGenesis，一个新颖的无监督语音合成框架，能够在没有监督的情况下发现潜在的说话人流形和有意义的声音编辑方向。VoxGenesis通过将高斯分布转化为由语义标记条件和对齐的语音分布，迫使模型学习与语义内容解耦的说话人分布。实验结果表明，VoxGenesis生成的说话人具有更高的多样性和真实感，并且潜在空间的操控能够产生一致且可识别的效果。

🔬 方法详解

问题定义：本论文旨在解决现有语音合成模型依赖有监督学习的问题，尤其是在情感、语调和说话风格等方面缺乏准确标签的挑战。

核心思路：VoxGenesis的核心思想是通过无监督学习发现潜在的说话人流形，利用高斯分布生成与语义内容解耦的说话人特征，从而实现灵活的声音编辑。

技术框架：VoxGenesis的整体架构包括高斯分布的生成、语音特征的条件对齐以及潜在空间的探索，主要模块包括特征提取、流形学习和声音合成。

关键创新：VoxGenesis的主要创新在于其无监督的学习方式和潜在空间的可操控性，使得生成的说话人具有独特的特征，这在以往方法中是无法实现的。

关键设计：在模型设计中，采用了特定的损失函数以确保生成的说话人特征与语义内容解耦，并通过高斯分布的采样实现多样化的声音生成。网络结构上，使用了深度神经网络来提取和映射语音特征。

🖼️ 关键图片

fig_0
fig_1
fig_2

📊 实验亮点

实验结果显示，VoxGenesis生成的说话人比以往方法具有更高的多样性和真实感，主观和客观评估均表明其性能显著提升。具体而言，VoxGenesis在生成特征的独特性和一致性方面表现优异，且潜在空间的操控效果可被人类识别，未对语音质量造成负面影响。

🎯 应用场景

VoxGenesis的研究成果在语音合成、虚拟助手、游戏音效和影视配音等领域具有广泛的应用潜力。通过无监督学习，能够生成多样化且个性化的声音，提升用户体验，并为未来的语音技术发展提供新的思路。

📄 摘要（原文）

Achieving nuanced and accurate emulation of human voice has been a longstanding goal in artificial intelligence. Although significant progress has been made in recent years, the mainstream of speech synthesis models still relies on supervised speaker modeling and explicit reference utterances. However, there are many aspects of human voice, such as emotion, intonation, and speaking style, for which it is hard to obtain accurate labels. In this paper, we propose VoxGenesis, a novel unsupervised speech synthesis framework that can discover a latent speaker manifold and meaningful voice editing directions without supervision. VoxGenesis is conceptually simple. Instead of mapping speech features to waveforms deterministically, VoxGenesis transforms a Gaussian distribution into speech distributions conditioned and aligned by semantic tokens. This forces the model to learn a speaker distribution disentangled from the semantic content. During the inference, sampling from the Gaussian distribution enables the creation of novel speakers with distinct characteristics. More importantly, the exploration of latent space uncovers human-interpretable directions associated with specific speaker characteristics such as gender attributes, pitch, tone, and emotion, allowing for voice editing by manipulating the latent codes along these identified directions. We conduct extensive experiments to evaluate the proposed VoxGenesis using both subjective and objective metrics, finding that it produces significantly more diverse and realistic speakers with distinct characteristics than the previous approaches. We also show that latent space manipulation produces consistent and human-identifiable effects that are not detrimental to the speech quality, which was not possible with previous approaches. Audio samples of VoxGenesis can be found at: \url{https://bit.ly/VoxGenesis}.