ResidencyRL: Reinforcement Learning in Simulated Clinical Environments
作者: Valentin Liévin, Samuel Schmidgall, Tim Strother, Alex Bijamov, Akshay Goel, Anil Palepu, Chunjong Park, Vahid Balazadeh, Min Woo Sun, Marius Guerard, Justin Chen, Dave Steiner, Vikram Dhillon, Ibrahim Azar, Akhil Mehta, Nicholas Spetsieris, Shilpan Shah, Maen Abdelrahim, Amit Dahiya, Yun Liu, Katherine Chou, Yossi Matias, Avinatan Hassidim, Dale R. Webster, Quoc V. Le, Raia Hadsell, Joelle Barral, Carey Radebaugh, Aleksandra Faust, Shekoofeh Azizi, Mike Schaekermann, Po-Hsuan Cameron Chen, Tao Tu, David Racz, Lin Yang
分类: cs.AI, cs.CL
发布日期: 2026-08-07
💡 一句话要点
提出ResidencyRL以优化临床决策过程中的强化学习
🎯 匹配领域: 支柱二：RL算法与架构 (RL & Architecture) 支柱九：具身大模型 (Embodied Foundation Models)
关键词: 强化学习 临床决策 人工智能 医学教育 多轮对话 大型语言模型 模拟训练
📋 核心要点
- 现有方法在优化临床决策序列方面仍显不足，尤其是在动态和复杂的临床环境中。
- ResidencyRL通过强化学习训练临床AI代理，利用模拟的多轮临床对话来提升决策能力。
- 实验结果显示，ResidencyRL代理在对抗条件下的诊断准确性提高了7.0%，漏诊红旗率降低了31%。
📝 摘要（中文）
在医学教育中，医生通过住院医师培训将学术知识转化为临床专业技能。现有的大型语言模型在静态医学基准上表现优异，但在优化临床决策序列方面仍显不足。本文提出ResidencyRL，一种通过模拟多轮临床接触（最多60轮对话和8次工具调用）训练临床人工智能代理的强化学习方法。ResidencyRL将策略代理与能够进行复杂对抗行为的大型语言模型模拟器相结合，训练过程中采用与诊断准确性、管理质量、沟通、文档记录和安全性相一致的结构化奖励。在独立评估中，ResidencyRL代理在对抗条件下的诊断准确性提高了7.0%，漏诊红旗率降低了31%。经过盲评的专家临床医生验证了这些提升，87.6%的对比中更倾向于训练后的代理。该代理在未见基准上表现优异，显示出强大的可迁移能力。
🔬 方法详解
问题定义：本文旨在解决现有临床决策优化方法不足的问题，尤其是在动态和复杂的临床环境中，现有方法无法有效处理多轮对话和不确定性。
核心思路：ResidencyRL通过强化学习训练临床AI代理，利用模拟的多轮临床对话，结合大型语言模型的能力，优化临床决策过程。设计上强调与真实临床环境的对抗性，以提高代理的适应性和决策质量。
技术框架：ResidencyRL的整体架构包括策略代理和大型语言模型模拟器，代理通过与模拟器的交互进行训练，采用结构化奖励机制来引导学习过程。主要模块包括对话生成、决策评估和奖励反馈。
关键创新：该研究的核心创新在于将强化学习与大型语言模型结合，形成了一种新的训练框架，能够有效模拟复杂的临床对话并优化决策过程。这一方法与传统的静态模型训练方法有本质区别。
关键设计：在设计中，采用了结构化奖励函数，重点关注诊断准确性、管理质量等指标。此外，模型的参数设置和网络结构经过精心调整，以确保在多轮对话中保持高效的学习和决策能力。
🖼️ 关键图片
📊 实验亮点
实验结果显示，ResidencyRL代理在对抗条件下的诊断准确性提高了7.0%（88.0%对比81.0%），漏诊红旗率降低了31%。在盲评中，87.6%的专家临床医生更倾向于使用训练后的代理，显示出显著的临床应用潜力。
🎯 应用场景
ResidencyRL的研究成果具有广泛的应用潜力，特别是在医学教育和临床培训领域。通过模拟真实的临床环境，该方法可以帮助医学生和住院医生提高临床决策能力，进而提升患者护理质量。未来，该技术有望在实际临床工作流程中得到验证和应用，推动医学教育的创新发展。
📄 摘要（原文）
In medical education, physicians convert academic knowledge into clinical expertise through residency: years of training across thousands of encounters, with diverse sources of feedback and progressively greater autonomy. Much of clinical reasoning relies on the patient encounter, a dialogue in which a clinician elicits history, refines diagnostic hypotheses, and decides management under uncertainty. While large language models (LLMs) excel on static medical benchmarks, methods to optimize the full sequence of clinical decisions remain underdeveloped. We present ResidencyRL, a reinforcement learning (RL) method for training clinical artificial intelligence (AI) agents through simulated multi-turn clinical encounters (up to 60 dialogue turns and 8 tool calls per trajectory). ResidencyRL pairs the policy agent with LLM simulators capable of complex, adversarial behaviors, training against a structured reward aligned to diagnostic accuracy, management quality, communication, documentation, and safety. On held-out evaluations, the ResidencyRL agent improves diagnostic accuracy by 7.0% under adversarial conditions (88.0% vs. 81.0%) and reduces missed red flag rates by 31%, demonstrating rigorous mitigation of premature closure. Blinded expert clinicians validated these gains, preferring the trained agent in 87.6% of side-by-side comparisons. The procedural competencies transfer to unseen benchmarks: the agent outperforms the base model across all six clinical axes of the AMIE multi-visit benchmark, and shows consistent directional improvements on AgentClinic and CRAFT-MD. Our findings demonstrate that sequential clinical decision-making can be effectively learned through multi-turn RL in simulation, yielding robust, generalizable capabilities, paving the way towards clinical mastery. Prospective validation with real-world workflows remains necessary to establish clinical utility.