Interpretable AI with Local Distillation
作者: Erin Craig, Yiling Huang, Snigdha Panigrahi
分类: stat.ME, cs.LG, stat.ML
发布日期: 2026-08-24
💡 一句话要点
提出局部蒸馏方法以提升AI模型的可解释性
🎯 匹配领域: 支柱二：RL算法与架构 (RL & Architecture) 支柱九：具身大模型 (Embodied Foundation Models)
关键词: 局部蒸馏 可解释性AI 线性建模 高斯随机化 特征选择 黑箱模型 机器学习
📋 核心要点
- 现有的高性能AI模型在预测准确性上优于传统方法，但缺乏可解释性，难以支持高风险决策。
- 本文提出局部蒸馏方法，通过黑箱教师模型指导线性学生模型，定义局部性并增强模型的可解释性。
- 在17个基准数据集上，局部蒸馏方法几乎达到了教师模型的准确性，同时生成了稀疏的线性模型，提升了可解释性。
📝 摘要（中文）
现代AI模型如表格基础模型和梯度提升集成模型在预测准确性上超越了传统方法，但缺乏可解释性。高风险决策需要既准确又可解释的模型。局部线性建模提供了一种解决方案，通过在每个查询点附近进行线性拟合来实现高准确性和透明性。本文提出局部蒸馏方法，利用黑箱“教师”模型指导正则化线性“学生”模型，定义局部性并通过高斯随机化增强解释能力。实验表明，该方法在17个基准数据集上几乎达到了教师模型的准确性，同时在每个测试点生成稀疏线性模型。
🔬 方法详解
问题定义：本文旨在解决现代AI模型在高风险决策中缺乏可解释性的问题。现有方法虽然在准确性上表现优异，但其黑箱特性使得用户难以理解模型的预测依据。
核心思路：提出局部蒸馏方法，通过黑箱教师模型指导线性学生模型，利用局部线性建模的思想，在每个查询点附近进行线性拟合，从而实现高准确性和可解释性。
技术框架：整体流程包括两个主要模块：首先，教师模型通过加权相似预测的训练样本定义局部性；其次，学生模型在每个查询点进行线性拟合，并通过高斯随机化增强模型的稳定性和解释能力。
关键创新：最重要的创新点在于通过局部蒸馏结合高斯随机化，能够在小扰动下保持特征选择概率的稳定性，这在现有方法中尚未实现。
关键设计：在损失函数中引入了正则化项，并通过加权教师模型的预测作为伪观察值来增强拟合的准确性。关键参数包括高斯随机化的幅度和教师模型的权重估计方法。
🖼️ 关键图片
📊 实验亮点
在17个基准数据集上，局部蒸馏方法的准确性几乎与教师模型相当，同时生成的稀疏线性模型在高维癌症基因表达数据中能够识别不同的患者亚组，显示出其在可解释性上的显著优势。
🎯 应用场景
该研究的潜在应用领域包括医疗决策支持、金融风险评估和任何需要高可解释性的AI系统。通过提升模型的可解释性，能够帮助决策者更好地理解模型的预测，从而做出更为明智的决策。未来，该方法有望在更多领域推广应用，促进AI技术的透明化和可信度提升。
📄 摘要（原文）
Modern AI models such as tabular foundation models and gradient-boosted ensembles can outpredict classical methods, but provide little basis for reasoning about their predictions. High-stakes decisions call for models that are both accurate and interpretable as built. Local linear modeling offers a path forward: a smooth regression function is locally well approximated by a linear one, allowing a linear fit near each query point to achieve high accuracy without sacrificing transparency. The challenges lie in learning what is "local" and developing statistical tools for interpretation. Here, we propose local distillation, in which a black-box "teacher" guides a regularized linear "student" model at each query point. The teacher (1) defines locality by upweighting training observations with similar predicted outcomes, and (2) anchors the fit with its prediction at the query point, included as a pseudo-observation whose weight is estimated from the data. For interpretation, we add a small amount of Gaussian randomization to the local objective and use refits to assess stability: selection frequencies identify reliable features at a query point, and clustering the randomized fits identifies stable subgroups across the data. Under the lasso penalty, we prove that this randomization yields feature-selection probabilities that are stable under small perturbations of the training responses. Across 17 benchmark datasets, local distillation nearly matches its AI teacher's accuracy while producing a sparse linear model at each test point. In a high-dimensional cancer gene expression example, the framework identifies patient subgroups whose local models use different genes; this heterogeneity is invisible to a global linear model, and difficult to surface in a black-box model.