Aligning GPTRec with Beyond-Accuracy Goals with Reinforcement Learning

📄 arXiv: 2403.04875v1 📥 PDF

作者: Aleksandr Petrov, Craig Macdonald

分类: cs.IR, cs.LG

发布日期: 2024-03-07

备注: Accepted by the 2nd Workshop The 2nd Workshop on Recommendation with Generative Models, in conjunction with The Web Conference 2024

💡 一句话要点

提出GPTRec以解决推荐系统中的多样性优化问题

🎯 匹配领域: 支柱二：RL算法与架构 (RL & Architecture)

关键词: 推荐系统 多样性优化 强化学习 Transformer模型 Next-K策略

📋 核心要点

  1. 现有的Top-K推荐模型在准确性上表现良好，但难以优化多样性等复杂指标，导致推荐系统的局限性。
  2. 本文提出了一种两阶段训练方法，首先使用教师-学生方法模仿Top-K模型，然后通过强化学习对GPTRec进行多样性优化。
  3. 实验结果显示，在4个测试案例中，GPTRec的Next-K生成方法在准确性和次要指标之间的权衡优于传统的贪婪重排序技术。

📝 摘要（中文）

现有的Transformer模型如BERT4Rec和SASRec在基于准确度的推荐任务中表现优异，但难以优化多样性等复杂指标。GPTRec模型采用Next-K策略，逐项生成推荐，能够更好地考虑项目间的复杂依赖关系。本文提出了一种两阶段训练方法，首先模仿传统Top-K模型的行为，然后通过强化学习对模型进行多样性和减少流行偏见的优化。实验表明，GPTRec在准确性与次要指标之间提供了更好的权衡。

🔬 方法详解

问题定义：本文旨在解决现有推荐系统在优化多样性等复杂指标时的不足，尤其是传统Top-K模型在这方面的局限性。

核心思路：提出一种两阶段训练方法，第一阶段模仿Top-K模型的行为，第二阶段通过强化学习优化多样性和减少流行偏见。

技术框架：整体流程分为两个阶段：第一阶段使用教师-学生方法训练GPTRec，第二阶段应用强化学习进行多样性优化。

关键创新：最重要的创新在于结合了Next-K生成策略与强化学习，使得模型能够更好地处理复杂的项目间依赖关系，超越传统的准确性优化。

关键设计：在训练过程中，设置了适当的损失函数以平衡准确性与多样性，并设计了特定的网络结构以支持逐项生成推荐。

🖼️ 关键图片

fig_0
fig_1
fig_2

📊 实验亮点

实验结果显示，在4个测试案例中，GPTRec的Next-K生成方法在3个案例中优于传统贪婪重排序技术，提供了更好的准确性与多样性之间的权衡，验证了其在复杂推荐场景中的有效性。

🎯 应用场景

该研究的潜在应用领域包括电商推荐、内容推荐和社交媒体推荐等，能够有效提升用户体验，增加推荐的多样性和相关性。未来，随着推荐系统对用户需求的理解不断深入，该方法有望在更广泛的场景中得到应用。

📄 摘要（原文）

Adaptations of Transformer models, such as BERT4Rec and SASRec, achieve state-of-the-art performance in the sequential recommendation task according to accuracy-based metrics, such as NDCG. These models treat items as tokens and then utilise a score-and-rank approach (Top-K strategy), where the model first computes item scores and then ranks them according to this score. While this approach works well for accuracy-based metrics, it is hard to use it for optimising more complex beyond-accuracy metrics such as diversity. Recently, the GPTRec model, which uses a different Next-K strategy, has been proposed as an alternative to the Top-K models. In contrast with traditional Top-K recommendations, Next-K generates recommendations item-by-item and, therefore, can account for complex item-to-item interdependencies important for the beyond-accuracy measures. However, the original GPTRec paper focused only on accuracy in experiments and needed to address how to optimise the model for complex beyond-accuracy metrics. Indeed, training GPTRec for beyond-accuracy goals is challenging because the interaction training data available for training recommender systems typically needs to be aligned with beyond-accuracy recommendation goals. To solve the misalignment problem, we train GPTRec using a 2-stage approach: in the first stage, we use a teacher-student approach to train GPTRec, mimicking the behaviour of traditional Top-K models; in the second stage, we use Reinforcement Learning to align the model for beyond-accuracy goals. In particular, we experiment with increasing recommendation diversity and reducing popularity bias. Our experiments on two datasets show that in 3 out of 4 cases, GPTRec's Next-K generation approach offers a better tradeoff between accuracy and secondary metrics than classic greedy re-ranking techniques.