Towards Optimizing Human-Centric Objectives in AI-Assisted Decision-Making With Offline Reinforcement Learning

📄 arXiv: 2403.05911v2 📥 PDF

作者: Zana Buçinca, Siddharth Swaroop, Amanda E. Paluch, Susan A. Murphy, Krzysztof Z. Gajos

分类: cs.HC, cs.AI

发布日期: 2024-03-09 (更新: 2024-04-14)

💡 一句话要点

提出离线强化学习以优化人本目标的AI决策支持

🎯 匹配领域: 支柱二：RL算法与架构 (RL & Architecture)

关键词: 离线强化学习 人机交互 决策支持 人本目标 技能提升 协作增强

📋 核心要点

  1. 现有AI决策支持工具主要关注决策准确性，未能充分考虑人本目标，如技能提升和协作增强。
  2. 本文提出利用离线强化学习建模人机决策过程，以优化人机交互，支持多样化的人本目标。
  3. 实验结果显示，优化后的策略在决策准确性和人机互补性上显著优于其他基线，且人类学习的优化较为复杂。

📝 摘要（中文）

设想AI决策支持工具不仅能提高决策准确性，还能提升技能、促进协作并增加任务乐趣。尽管优化人本目标的潜力巨大，现有AI工具设计仍主要集中于决策准确性。本文提出离线强化学习作为一种通用方法，旨在建模人机决策过程，以优化人机交互。通过对人机交互数据的学习，我们实现了针对决策任务的准确性和人类学习的优化策略。实验结果表明，使用优化策略的人在决策准确性和人机互补性方面显著优于其他支持类型，同时人类学习的优化较为困难，学习提升并不总是显著。我们的研究强调了在AI辅助决策中考虑人本目标的重要性，开启了优化人机交互的新研究挑战。

🔬 方法详解

问题定义：本文旨在解决现有AI决策支持工具仅关注决策准确性的问题，忽视了人本目标的优化，如技能提升和协作。

核心思路：提出离线强化学习作为建模人机决策的通用方法，通过分析历史人机交互数据，优化决策支持策略，以实现多样化的人本目标。

技术框架：整体框架包括数据收集、策略学习和策略评估三个主要模块。首先收集人机交互数据，然后利用离线强化学习算法学习优化策略，最后通过实验评估策略的有效性。

关键创新：最重要的创新在于将离线强化学习应用于人机决策建模，突破了传统方法仅关注准确性的局限，能够同时优化人类学习和决策支持。

关键设计：在策略学习中，采用特定的损失函数以平衡决策准确性与人类学习的优化，网络结构设计上考虑了人机交互的动态特性，以适应不同用户的需求。

📊 实验亮点

实验结果表明，参与者在使用优化策略的情况下，决策准确性提高了显著的幅度，且在人机互补性方面表现优于其他支持类型。具体而言，优化后的策略在两个实验中（N=316和N=964）均显示出明显的性能提升，尤其是在决策准确性方面。

🎯 应用场景

该研究的潜在应用领域包括医疗决策支持、教育技术和智能助手等。通过优化人机交互，AI工具能够更好地满足用户需求，提升决策质量和用户体验，未来可能在各行业中广泛应用，推动人机协作的进步。

📄 摘要（原文）

Imagine if AI decision-support tools not only complemented our ability to make accurate decisions, but also improved our skills, boosted collaboration, and elevated the joy we derive from our tasks. Despite the potential to optimize a broad spectrum of such human-centric objectives, the design of current AI tools remains focused on decision accuracy alone. We propose offline reinforcement learning (RL) as a general approach for modeling human-AI decision-making to optimize human-AI interaction for diverse objectives. RL can optimize such objectives by tailoring decision support, providing the right type of assistance to the right person at the right time. We instantiated our approach with two objectives: human-AI accuracy on the decision-making task and human learning about the task and learned decision support policies from previous human-AI interaction data. We compared the optimized policies against several baselines in AI-assisted decision-making. Across two experiments (N=316 and N=964), our results demonstrated that people interacting with policies optimized for accuracy achieve significantly better accuracy -- and even human-AI complementarity -- compared to those interacting with any other type of AI support. Our results further indicated that human learning was more difficult to optimize than accuracy, with participants who interacted with learning-optimized policies showing significant learning improvement only at times. Our research (1) demonstrates offline RL to be a promising approach to model human-AI decision-making, leading to policies that may optimize human-centric objectives and provide novel insights about the AI-assisted decision-making space, and (2) emphasizes the importance of considering human-centric objectives beyond decision accuracy in AI-assisted decision-making, opening up the novel research challenge of optimizing human-AI interaction for such objectives.