Offline Reinforcement Learning for Hemodynamic Management of Sepsis in the ICU: a MIMIC-IV Study with Dual Off-Policy Evaluation

📄 arXiv: 2608.16482v1 📥 PDF

作者: Marc Pérez-Roig, David Fernández-Narro, Carlos Sáez

分类: cs.AI, cs.LG

发布日期: 2026-08-17

💡 一句话要点

提出离线强化学习以优化ICU中脓毒症的血流动力学管理

🎯 匹配领域: 支柱二：RL算法与架构 (RL & Architecture)

关键词: 脓毒症管理 强化学习 离线评估 临床决策支持 重症监护 随机森林 马尔可夫决策过程

📋 核心要点

  1. 现有方法在脓毒症管理中依赖临床判断，缺乏系统性和可靠性，导致治疗效果不稳定。
  2. 本研究提出了一种结合离线估计与可靠性分析的框架，以评估脓毒症治疗策略的有效性。
  3. 实验结果表明，学习到的策略在临床实践中具有可行性，且在静脉输液使用上有所减少，表现优于临床医生的决策。

📝 摘要（中文）

脓毒症患者的静脉输液和血管收缩药物的剂量调整是在不确定性下进行的序列决策，主要依赖临床判断，因此成为了从历史护理中应用强化学习的自然目标。由于学习到的策略无法在患者身上进行试验，其价值必须通过离线评估来估计，而这种估计可能脆弱且过于乐观。本研究通过结合离线估计、可靠性诊断和临床一致性分析，提出了一个透明的验证框架，以可靠地评估脓毒症治疗策略。我们在MIMIC-IV数据库中对36872例脓毒症ICU住院病例进行了建模，采用离散化的马尔可夫决策过程，解决了流体和血管收缩药物的剂量问题。

🔬 方法详解

问题定义：本研究旨在解决脓毒症患者在ICU中静脉输液和血管收缩药物剂量调整的决策问题。现有方法主要依赖临床判断，缺乏系统性评估，导致治疗效果不稳定。

核心思路：通过离线强化学习方法，从历史数据中学习治疗策略，并结合离线评估和可靠性分析，确保策略的有效性和临床适用性。

技术框架：整体架构包括数据建模、策略学习和评估三个主要模块。首先，从MIMIC-IV数据库中提取36872例脓毒症病例数据，构建离散化的马尔可夫决策过程。然后，利用随机森林估计临床行为策略，最后通过加权重要性采样和拟合Q评估对学习到的策略进行评估。

关键创新：本研究的创新点在于结合了离线估计、可靠性诊断和临床一致性分析，提供了一种透明的验证框架，克服了传统方法中估计不稳定的问题。

关键设计：在模型构建中，定义了1000个状态和25个动作，采用五乘五的流体和血管收缩药物水平网格。使用随机森林来估计临床行为策略，避免了有效样本量的崩溃，确保了重要性采样估计的稳定性。

🖼️ 关键图片

fig_0
fig_1
fig_2

📊 实验亮点

实验结果显示，学习到的策略在加权重要性采样下的回报为50.8，拟合Q评估下为46.8，均高于临床医生的38.2，且与观察到的实践偏差仅为0.18，表明该策略在临床上具有可行性和改进潜力。

🎯 应用场景

该研究的潜在应用领域包括重症监护和脓毒症管理，能够为临床医生提供基于数据的决策支持，优化治疗方案，提高患者的生存率和治疗效果。未来，该方法有望推广至其他急性疾病的管理中，提升整体医疗质量。

📄 摘要（原文）

The dosing of intravenous fluids and vasopressors in sepsis is a sequential decision made under uncertainty and guided largely by clinical judgment, which makes it a natural target for reinforcement learning from historical care. Because a learned policy cannot be trialed on patients, its value must be estimated off-policy, and such estimates can be fragile and optimistic. This work advances the reliable evaluation of sepsis treatment policies by combining off-policy estimation, reliability diagnostics, and clinician-agreement analyses in a transparent validation framework. We modeled fluid and vasopressor dosing on a cohort of 36,872 septic ICU stays drawn from the MIMIC-IV critical-care database, as a discretized Markov decision process with 1,000 states and 25 actions, defined by a five-by-five grid of fluid and vasopressor levels and solved by policy iteration. The clinicians' behavior policy was estimated with a random forest, which mitigated the collapse of the Effective Sample Size (ESS 50.1 against 4.0 with smoothed counts) that otherwise destabilizes the importance-sampling estimate. The learned policy was evaluated with two estimators, weighted importance sampling (WIS) and fitted Q evaluation (FQE), with the ESS and clinician agreement as reliability checks. An empirical variable selection found that the composition of the state matters more than its size. Both estimators place the learned policy above the clinicians' return (WIS 50.8 and FQE 46.8 against 38.2, ESS 50.1), yet it departs only modestly from observed practice (total variation 0.18), favoring less intravenous fluid. These retrospective single-center off-policy results support the learned policy as a clinically plausible refinement of observed practice and motivate its further evaluation as a discordance-based clinical decision-support approach.