Beyond Borrowed Histories: Person-Aligned User Simulation for Interactive Role-Playing Evaluation

📄 arXiv: 2607.27816v1 📥 PDF

作者: Yuhang Zhu, Mingxuan Du, Benfeng Xu, Jie Gao, Lingyun Yu, Hongtao Xie

分类: cs.CL, cs.AI

发布日期: 2026-07-30

备注: 29 pages, 3 figures, including supplementary material. Resources: https://github.com/Zhuyh1139/PALATE

💡 一句话要点

提出PALATE以解决角色扮演代理评估中的用户体验问题

🎯 匹配领域: 支柱九：具身大模型 (Embodied Foundation Models)

关键词: 角色扮演代理 用户模拟 个性化评估 多轮对话 情感支持 大型语言模型

📋 核心要点

  1. 现有评估方法依赖固定对话历史，无法科学地评估RPA在真实多轮对话中的表现。
  2. 本文提出PALATE，通过用户模拟器进行个性化评估，解决了用户体验的多样性问题。
  3. 实验结果表明，个性化评估标准与人类判断的一致性高于传统通用标准，提升了评估的准确性。

📝 摘要（中文）

角色扮演代理（RPA）已成为大型语言模型的重要应用，用户通过与RPA进行多轮对话获得情感支持，因此可靠的评估方法至关重要。现有基准通常要求RPA基于固定对话历史进行评估，存在两大局限：一是无法科学评估RPA在真实多轮对话中的角色扮演能力，二是用户体验因人而异，固定评估标准未必符合用户满意度。为此，本文提出PALATE（个性化用户模拟评估），构建了基于用户模拟器的可扩展RPA基准，配备300个角色档案，通过个性化评估标准提高了与人类判断的一致性。

🔬 方法详解

问题定义：本文旨在解决现有角色扮演代理评估方法的局限性，特别是固定对话历史和通用评估标准无法反映用户真实体验的问题。

核心思路：提出PALATE，通过构建用户模拟器，允许RPA在多轮对话中与个性化角色档案进行互动，从而实现更科学的评估。

技术框架：PALATE的整体架构包括用户模拟器、角色档案库和个性化评估标准。用户模拟器根据不同角色档案生成多轮对话，评估RPA的表现。

关键创新：PALATE的主要创新在于引入个性化评估标准，能够根据用户的具体需求和偏好进行评估，而不是依赖于固定的通用标准。

关键设计：在设计中，PALATE使用了300个角色档案，训练了五个用户模拟器，并结合通用质量标准和个性化标准进行评估，确保评估结果的多样性和准确性。

🖼️ 关键图片

img_0
img_1
img_2

📊 实验亮点

在对16个候选RPA的评估中，PALATE能够分别表征通用对话质量、长时间会话能力和个性化用户体验，个性化评估标准与人类判断的一致性显著高于通用标准，提升幅度达到未知。

🎯 应用场景

该研究的潜在应用领域包括情感支持系统、虚拟助手和游戏角色扮演等。通过提供更准确的评估方法，PALATE能够帮助开发者优化RPA的用户体验，提升用户满意度，推动相关技术的进一步发展。

📄 摘要（原文）

Role-playing agents (RPAs) have become one of the most important consumer applications of large language models. Users engage in multi-turn conversations with RPAs for experiences such as emotional comfort, making reliable evaluation essential for measuring capability, comparing systems, and guiding further improvement. Existing benchmarks, however, typically require an RPA to continue a fixed dialogue history and then evaluate the continuation using a fixed rubric detached from the user. We identify and empirically demonstrate two limitations of this design. First, an RPA's output is shaped by the preceding dialogue history, preventing a scientifically grounded assessment of its role-playing ability in real multi-turn settings. Second, user experience varies substantially across individuals, and conventional fixed rubrics need not align with user satisfaction. We therefore introduce PALATE (Person-Aligned LLM-Simulated-User Assessment with Tailored Evaluation), a scalable RPA benchmark built on user simulators. PALATE is accompanied by a pool of 300 character profiles. Its main evaluation trains five per-user simulators and lets them engage candidate RPAs in free-form, multi-turn conversations over a pre-frozen panel of character profiles. Alongside a general quality rubric, we construct personalized rubrics to measure user satisfaction; on held-out annotated data, the personalized rubrics show higher agreement with human judgments than the general rubric. In the main evaluation of 16 candidates, PALATE separately characterizes generic turn quality, long-horizon session capability, and per-user experience on multi-turn trajectories co-constructed by each candidate. It thereby produces interpretable evaluations of specific user-RPA pairs rather than compressing systems into a single user-independent ranking.