Distill on a Diet: Efficient Knowledge Distillation via Learnable Data Pruning

📄 arXiv: 2606.25488v1 📥 PDF

作者: Yifan Wu, Yiqi Wang, Xichen Ye, Wenjing Yan, Xiaoqiang Li, Cheng Jin, Xiangyu Yue, Weizhong Zhang

分类: cs.LG

发布日期: 2026-06-24

备注: Acceepted by ECCV 2026

💡 一句话要点

提出IF-Beta框架以提高知识蒸馏中的数据剪枝效率

🎯 匹配领域: 支柱二：RL算法与架构 (RL & Architecture)

关键词: 知识蒸馏 数据剪枝 影响函数 可学习策略 深度学习

📋 核心要点

  1. 现有知识蒸馏方法在数据剪枝时未能有效降低计算开销，导致效率低下。
  2. 本文提出IF-Beta框架，通过影响函数和可学习采样策略优化数据剪枝过程，提升蒸馏效果。
  3. 实验结果表明，IF-Beta在多种剪枝比例下均优于其他基线，且在少量数据和计算下训练的学生模型性能更佳。

📝 摘要（中文）

知识蒸馏（KD）广泛用于在资源受限环境中获取紧凑模型。然而，蒸馏过程的计算开销常被忽视，提出通过数据剪枝减少数据和计算量以获得更好的学生模型。现有数据剪枝方法未针对KD设计，导致效率低下。为此，本文提出IF-Beta，一个结合影响函数与可学习采样策略的高效数据剪枝框架。通过实验证明，IF-Beta在CIFAR-10/100和ImageNet上表现优异，能够在较少数据和计算下超越全数据集蒸馏的学生模型。

🔬 方法详解

问题定义：本文解决知识蒸馏过程中数据剪枝效率低下的问题。现有方法在剪枝时引入了较大的计算开销，且未能有效捕捉KD所需的样本特性。

核心思路：提出IF-Beta框架，结合影响函数作为样本影响的有效估计器，并通过Beta分布参数化可学习的采样策略，以适应不同的剪枝需求。

技术框架：IF-Beta的整体架构包括两个主要环节：内层循环在教师特征空间中进行快速代理训练，外层循环则更新采样策略参数以最大化蒸馏性能。

关键创新：IF-Beta的核心创新在于将影响函数与可学习采样策略结合，形成了一个高效的剪枝框架，显著提升了蒸馏性能。与现有方法相比，IF-Beta避免了固定启发式选择规则的局限性。

关键设计：IF-Beta采用Beta分布进行采样策略参数化，设计了双层优化目标，内层目标与KD对齐，确保了快速的代理训练和高效的策略更新。

🖼️ 关键图片

fig_0
fig_1
fig_2

📊 实验亮点

实验结果显示，IF-Beta在CIFAR-10/100和ImageNet数据集上均优于其他基线方法，尤其在高达50%的剪枝比例下，学生模型的性能超越了使用全数据集蒸馏的模型，展现出显著的性能提升。

🎯 应用场景

该研究具有广泛的应用潜力，尤其在资源受限的设备上，如移动端和嵌入式系统中，能够有效提升模型的推理效率和性能。未来，IF-Beta框架可扩展至其他机器学习任务，推动知识蒸馏技术的进一步发展。

📄 摘要（原文）

Knowledge Distillation (KD) is widely used to obtain compact models for efficient inference in resource-constrained environments. Yet the computational overhead of the distillation process itself is often overlooked, raising the question of whether a better student model can be obtained with less data and less compute via data pruning. However, existing data pruning methods are not designed for KD: some introduce substantial overhead, such as obtaining training dynamics through retraining, while others rely on heuristic selection rules that fail to capture what KD actually requires, often resulting in suboptimal subsets. To address these issues, we propose IF-Beta, an efficient data pruning framework that combines influence functions with a learnable sampling policy. Empirically, we first demonstrate that influence functions can serve as an effective and efficient estimator of sample impact in KD settings, where only a pretrained teacher is available. Building on this, our sampling policy is specifically parameterized by a Beta distribution, whose highly flexible two-parameter family allows the policy to adapt to diverse pruning regimes rather than being tied to fixed heuristic forms. Next, we formulate KD pruning as optimizing this policy through a bilevel objective, where the inner loop operates in the teacher feature space with a KD-aligned objective, enabling fast proxy training, while the outer loop updates the policy parameters to maximize distillation performance. This design ensures that IF-Beta is both computationally efficient and inherently aligned with the goals of KD. Extensive experiments on CIFAR-10/100 and ImageNet show that IF-Beta consistently outperforms other baselines across a wide range of pruning ratios. Remarkably, IF-Beta enables students trained on less data and less compute to surpass the performance of students distilled on the full dataset.