Sequential Subset Matching for Dataset Distillation

📄 arXiv: 2311.01570v1 📥 PDF

作者: Jiawei Du, Qin Shi, Joey Tianyi Zhou

分类: cs.CV, cs.LG

发布日期: 2023-11-02

🔗 代码/项目: GITHUB

💡 一句话要点

提出SeqMatch以解决数据蒸馏中的耦合问题

🎯 匹配领域: 支柱二：RL算法与架构 (RL & Architecture)

关键词: 数据集蒸馏 深度学习 合成数据 优化策略 特征提取

📋 核心要点

  1. 现有数据集蒸馏方法将合成数据集视为统一实体，静态优化导致耦合问题，影响性能。
  2. 本文提出SeqMatch策略，通过顺序生成合成实例，自适应优化数据，解决耦合问题。
  3. SeqMatch在SVNH、CIFAR-10、CIFAR-100和Tiny ImageNet等多个数据集上表现优异，超越了现有方法。

📝 摘要（中文）

数据集蒸馏是一项新兴任务，旨在合成小规模数据集以降低数据存储和模型训练成本。现有方法将整个合成数据集视为统一实体，导致静态优化可能引发耦合问题，从而影响高层特征的提取。本文提出了一种新的数据集蒸馏策略SeqMatch，通过自适应优化合成数据，鼓励知识的顺序获取，从而显著提升性能。实验结果表明，SeqMatch在多个数据集上超过了现有最先进的方法。

🔬 方法详解

问题定义：本文解决的是数据集蒸馏中的耦合问题，现有方法在优化合成数据时未考虑数据间的相互影响，导致性能下降。

核心思路：SeqMatch通过顺序生成合成实例，逐步优化数据集，鼓励知识的顺序获取，从而有效避免耦合问题。

技术框架：SeqMatch的整体架构包括数据生成模块和优化模块，前者负责生成合成实例，后者则根据生成的实例进行自适应优化。

关键创新：SeqMatch的核心创新在于其动态优化策略，区别于传统方法的静态优化，能够更好地提取高层特征。

关键设计：在SeqMatch中，采用了特定的损失函数来评估合成实例的质量，并通过调整生成策略来优化合成数据的多样性和代表性。

🖼️ 关键图片

fig_0
fig_1
fig_2

📊 实验亮点

SeqMatch在多个数据集上表现出色，尤其是在SVNH、CIFAR-10、CIFAR-100和Tiny ImageNet上，相较于现有最先进的方法，性能提升显著，具体提升幅度未知，展示了其在数据集蒸馏领域的优越性。

🎯 应用场景

该研究的潜在应用领域包括深度学习模型的训练和优化，尤其是在数据稀缺或存储成本高的情况下。SeqMatch可以帮助研究人员和工程师更高效地利用有限的数据资源，提升模型的学习效果和泛化能力，具有重要的实际价值和未来影响。

📄 摘要（原文）

Dataset distillation is a newly emerging task that synthesizes a small-size dataset used in training deep neural networks (DNNs) for reducing data storage and model training costs. The synthetic datasets are expected to capture the essence of the knowledge contained in real-world datasets such that the former yields a similar performance as the latter. Recent advancements in distillation methods have produced notable improvements in generating synthetic datasets. However, current state-of-the-art methods treat the entire synthetic dataset as a unified entity and optimize each synthetic instance equally. This static optimization approach may lead to performance degradation in dataset distillation. Specifically, we argue that static optimization can give rise to a coupling issue within the synthetic data, particularly when a larger amount of synthetic data is being optimized. This coupling issue, in turn, leads to the failure of the distilled dataset to extract the high-level features learned by the deep neural network (DNN) in the latter epochs. In this study, we propose a new dataset distillation strategy called Sequential Subset Matching (SeqMatch), which tackles this problem by adaptively optimizing the synthetic data to encourage sequential acquisition of knowledge during dataset distillation. Our analysis indicates that SeqMatch effectively addresses the coupling issue by sequentially generating the synthetic instances, thereby enhancing its performance significantly. Our proposed SeqMatch outperforms state-of-the-art methods in various datasets, including SVNH, CIFAR-10, CIFAR-100, and Tiny ImageNet. Our code is available at https://github.com/shqii1j/seqmatch.