Reward Design for Justifiable Sequential Decision-Making
作者: Aleksa Sukovic, Goran Radanovic
分类: cs.LG, cs.AI
发布日期: 2024-02-24
💡 一句话要点
提出基于辩论的奖励模型以增强决策的可辩护性
🎯 匹配领域: 支柱二：RL算法与架构 (RL & Architecture)
关键词: 可辩护决策 强化学习 辩论模型 医疗决策 多代理系统 人类评估 策略学习
📋 核心要点
- 现有方法在高风险决策场景中缺乏有效的可辩护性，难以满足人类期望和社会规范。
- 本文提出通过辩论游戏的奖励模型来训练强化学习代理，使其决策过程具有更强的可辩护性。
- 实验结果显示，基于辩论的反馈显著提升了决策的可辩护性，并与理想评估者的反馈相当。
📝 摘要（中文）
为代理赋予使用支持证据来证明决策的能力是负责任决策的基石。确保这些证明符合人类期望和社会规范尤为重要，尤其是在医疗等高风险场景中。本文提出了一种基于辩论的奖励模型，用于强化学习代理，其中零和辩论游戏的结果量化了特定状态下决策的可辩护性。通过这种奖励模型训练的可辩护策略，其决策更易于用支持证据进行验证。我们展示了该方法在为脓毒症患者开处方和证明治疗决策中的潜力。实验表明，增强奖励的辩论反馈信号使得政策在评估中更受青睐，同时几乎不牺牲性能。
🔬 方法详解
问题定义：本文旨在解决强化学习代理在高风险决策中缺乏可辩护性的问题。现有方法往往无法提供足够的支持证据，导致决策难以被验证和接受。
核心思路：论文提出了一种基于辩论的奖励模型，通过模拟辩论过程来量化决策的可辩护性。代理在辩论中提供支持证据，最终由人类评估者判断哪一决策更具说服力。
技术框架：整体架构包括两个主要模块：辩论代理和评估代理。辩论代理负责提出支持证据，而评估代理则根据辩论结果给出奖励信号，指导策略学习。
关键创新：最重要的技术创新在于引入辩论机制作为奖励信号，这与传统的环境奖励机制有本质区别，能够更好地反映决策的可辩护性。
关键设计：在模型设计中，采用了零和游戏的结构，设置了适当的损失函数以平衡辩论的公平性和有效性，同时确保代理能够生成抵御反驳的证据。
🖼️ 关键图片
📊 实验亮点
实验结果表明，使用辩论反馈的策略在评估中获得了更高的认可度，相较于仅依赖环境奖励的策略，表现提升显著，且在整体性能上几乎没有损失。这表明辩论机制有效地提取了决策评估中最相关的信息。
🎯 应用场景
该研究的潜在应用领域包括医疗决策支持系统、法律判决辅助工具以及任何需要高可辩护性决策的领域。通过增强决策的透明度和可解释性，未来可能会提高人们对自动化系统的信任度，促进其在关键领域的应用。
📄 摘要（原文）
Equipping agents with the capacity to justify made decisions using supporting evidence represents a cornerstone of accountable decision-making. Furthermore, ensuring that justifications are in line with human expectations and societal norms is vital, especially in high-stakes situations such as healthcare. In this work, we propose the use of a debate-based reward model for reinforcement learning agents, where the outcome of a zero-sum debate game quantifies the justifiability of a decision in a particular state. This reward model is then used to train a justifiable policy, whose decisions can be more easily corroborated with supporting evidence. In the debate game, two argumentative agents take turns providing supporting evidence for two competing decisions. Given the proposed evidence, a proxy of a human judge evaluates which decision is better justified. We demonstrate the potential of our approach in learning policies for prescribing and justifying treatment decisions of septic patients. We show that augmenting the reward with the feedback signal generated by the debate-based reward model yields policies highly favored by the judge when compared to the policy obtained solely from the environment rewards, while hardly sacrificing any performance. Moreover, in terms of the overall performance and justifiability of trained policies, the debate-based feedback is comparable to the feedback obtained from an ideal judge proxy that evaluates decisions using the full information encoded in the state. This suggests that the debate game outputs key information contained in states that is most relevant for evaluating decisions, which in turn substantiates the practicality of combining our approach with human-in-the-loop evaluations. Lastly, we showcase that agents trained via multi-agent debate learn to propose evidence that is resilient to refutations and closely aligns with human preferences.