RLHF from Heterogeneous Feedback via Personalization and Preference Aggregation

📄 arXiv: 2405.00254v2 📥 PDF

作者: Chanwoo Park, Mingyang Liu, Dingwen Kong, Kaiqing Zhang, Asuman Ozdaglar

分类: cs.AI, cs.LG

发布日期: 2024-04-30 (更新: 2024-05-27)

备注: Added experiments

💡 一句话要点

提出个性化与偏好聚合方法以解决人类反馈异质性问题

🎯 匹配领域: 支柱二：RL算法与架构 (RL & Architecture)

关键词: 强化学习 人类反馈 偏好聚合 个性化学习 机制设计 模型对齐 样本复杂度

📋 核心要点

  1. 现有RLHF方法假设人类偏好同质，无法有效处理偏好异质性和反馈的战略性行为。
  2. 提出个性化和聚合两种框架，通过学习多个奖励模型和聚合真实偏好来解决反馈异质性问题。
  3. 通过样本复杂度保证和机制设计，确保了反馈的真实性和社会福利的最大化，提升了模型的对齐效果。

📝 摘要（中文）

强化学习从人类反馈（RLHF）是一种有效的技术，用于使人工智能系统与人类价值观对齐。现有的RLHF方法通常假设人类偏好相对同质，能够通过单一奖励模型进行编码。本文针对人类偏好的固有异质性及其在反馈中可能表现出的战略行为，提出了两种框架：基于个性化的方法和基于聚合的方法。前者通过表示学习和聚类学习多个奖励模型，以平衡偏差和方差；后者则在单模型框架下，通过奖励和偏好聚合的方法，确保真实偏好的汇总，并处理可能的战略性反馈。我们的研究为RLHF提供了新的视角和方法论。

🔬 方法详解

问题定义：本文解决的是人类反馈的异质性问题，现有方法往往假设人类偏好是同质的，导致模型无法有效应对多样化的反馈。

核心思路：提出个性化和聚合两种方法，前者通过学习多个奖励模型来应对偏好异质性，后者则在单一模型框架下聚合真实偏好，确保反馈的真实性。

技术框架：整体框架包括两个主要模块：个性化模块通过表示学习和聚类学习多个奖励模型，聚合模块则通过奖励和偏好聚合方法整合反馈。

关键创新：最重要的创新在于通过个性化学习多个奖励模型，平衡偏差和方差，同时在聚合方法中引入机制设计以确保真实反馈的汇总。

关键设计：在个性化方法中，采用了表示学习和聚类技术，确保模型能够适应不同的偏好；在聚合方法中，利用效用主义和Leximin方法进行奖励聚合，并设计了处理战略性反馈的机制。

📊 实验亮点

实验结果表明，提出的方法在处理异质性反馈时，相较于传统RLHF方法，模型的对齐效果显著提升，样本复杂度得到了有效控制，确保了反馈的真实性和社会福利的最大化。

🎯 应用场景

该研究在人工智能系统的训练和优化中具有广泛的应用潜力，尤其是在需要人类反馈的领域，如自然语言处理、推荐系统和人机交互等。通过更好地理解和整合人类反馈，未来的AI系统能够更有效地对齐人类价值观，提升用户体验和满意度。

📄 摘要（原文）

Reinforcement learning from human feedback (RLHF) has been an effective technique for aligning AI systems with human values, with remarkable successes in fine-tuning large-language models recently. Most existing RLHF paradigms make the underlying assumption that human preferences are relatively homogeneous, and can be encoded by a single reward model. In this paper, we focus on addressing the issues due to the inherent heterogeneity in human preferences, as well as their potential strategic behavior in providing feedback. Specifically, we propose two frameworks to address heterogeneous human feedback in principled ways: personalization-based one and aggregation-based one. For the former, we propose two approaches based on representation learning and clustering, respectively, for learning multiple reward models that trades off the bias (due to preference heterogeneity) and variance (due to the use of fewer data for learning each model by personalization). We then establish sample complexity guarantees for both approaches. For the latter, we aim to adhere to the single-model framework, as already deployed in the current RLHF paradigm, by carefully aggregating diverse and truthful preferences from humans. We propose two approaches based on reward and preference aggregation, respectively: the former utilizes both utilitarianism and Leximin approaches to aggregate individual reward models, with sample complexity guarantees; the latter directly aggregates the human feedback in the form of probabilistic opinions. Under the probabilistic-opinion-feedback model, we also develop an approach to handle strategic human labelers who may bias and manipulate the aggregated preferences with untruthful feedback. Based on the ideas in mechanism design, our approach ensures truthful preference reporting, with the induced aggregation rule maximizing social welfare functions.