Differentiating Through Dual Prices: End-to-End Policy Learning Under Capacity Constraints
作者: Mohammadsaeed Haghi, Mahdi Salmani, Nima Kelidari
分类: cs.LG
发布日期: 2026-08-05
备注: 15 pages, 7 figures, 2 algorithms. Includes a technical appendix with full proofs, an excess-value decomposition, ablations, and reproducibility details. Code: https://github.com/mahdisalmani/end2end-capacity-constrained-policy-learning
💡 一句话要点
提出双价格区分方法以解决资源分配中的容量约束问题
🎯 匹配领域: 支柱二：RL算法与架构 (RL & Architecture)
关键词: 资源分配 容量约束 端到端训练 双价格 政策学习 社会服务优化 排队系统
📋 核心要点
- 现有方法在资源分配中存在决策盲目性，无法有效处理容量约束，导致资源利用不均。
- 本文提出通过双价格进行端到端训练，直接优化已部署策略的价值，克服传统方法的局限性。
- 在六个数据集的实验中，端到端方法在所有延迟成本下均表现优异，尤其在资源紧张时显著减少排队延迟。
📝 摘要（中文）
许多社会服务在资源稀缺的情况下，需要对逐个到达的个体进行即时决策，同时确保长期资源使用不超过其容量。本文研究如何从观察数据中学习这种分配策略。传统方法通常是决策盲目的，通过回归为每个选项拟合一个结果模型，并根据预测结果和价格进行分配。我们提出了一种端到端的训练方法，通过双价格对已部署策略的价值进行反向传播。我们研究了两种模型：一种精确的非凸模型和一种凸松弛模型，后者在期望上始终满足容量约束。实验结果表明，端到端方法在多个数据集上表现优异，尤其在资源紧张的情况下，显著提高了政策价值。
🔬 方法详解
问题定义：本文旨在解决在容量约束下的资源分配问题，现有方法由于决策盲目性，常常导致容量超限和资源浪费。
核心思路：我们提出通过双价格进行端到端的训练，利用反向传播优化已部署策略的价值，从而实现更有效的资源分配。
技术框架：整体方法包括两个主要阶段：首先，构建结果模型并进行端到端训练；其次，通过双价格进行策略优化，确保在容量约束下进行有效分配。
关键创新：最重要的创新在于通过双价格实现对已部署策略的价值进行反向传播，这一方法与传统的决策盲目方法本质上不同，能够更好地适应资源稀缺的场景。
关键设计：在模型设计中，我们采用了非凸和凸松弛两种形式，确保在期望上满足容量约束，并通过调节平滑温度和手臂数量的对数来控制模型的最优性。具体的损失函数和网络结构设计也经过精心调整，以适应不同的数据集和资源配置。
🖼️ 关键图片
📊 实验亮点
实验结果显示，端到端训练方法在六个数据集上均取得了最佳的部署调整价值指数，尤其在容量紧张的情况下，决策盲目基线常常违反容量限制，导致排队延迟显著增加。在最大的数据集上，端到端训练的政策价值显著高于容量匹配的神经基线，显示出其优越性。
🎯 应用场景
该研究的潜在应用领域包括医疗资源分配、社会服务优化等场景，能够在资源稀缺的情况下实现更高效的决策，提升服务质量和资源利用率。未来，该方法有望推广至更多需要实时决策的领域，推动智能决策系统的发展。
📄 摘要（原文）
Many social services assign scarce resources, such as housing assistance or hospital interventions, to people who arrive one at a time: each arrival must receive a decision immediately, and the long-run usage of every resource must stay within its capacity. We study how to learn such an assignment policy from logged observational data. The standard pipeline is decision-blind: fit one outcome model per arm by regression, price each capacitated resource from the fitted models, and assign each arrival the arm whose predicted outcome minus price is largest. We instead train the outcome models end-to-end, differentiating an off-policy estimate of the deployed policy's value through the dual prices themselves. We study two formulations: an exact nonconvex one, and a convex relaxation whose optimum always satisfies the capacity constraints in expectation and which is suboptimal by at most a term linear in the smoothing temperature and logarithmic in the number of arms. Every method is evaluated in a queueing simulation with resources replenished at their capacity rates. Across six datasets, the two end-to-end variants take the top slots on a deployment-adjusted value index at every delay cost, including zero; when capacities are binding, decision-blind baselines frequently violate them and incur much longer queueing delays. On the largest dataset, a hospital cohort of seventy thousand patients, end-to-end training also achieves significantly higher policy value, a margin that survives a capacity-matched neural baseline. Flexible decision-blind regression remains the stronger pure predictor where ground truth is measurable; end-to-end training is best suited to settings where resources are genuinely scarce and feasibility matters.