When Preferences Fail to Become Incentives: A Utility-Behavior Gap in Large Language Models
作者: Yujun Zhou, Christopher M. Ackerman
分类: cs.AI
发布日期: 2026-06-22
备注: 23 pages, 9 figures, 11 tables
💡 一句话要点
探讨偏好与激励之间的差距以提升大语言模型行为理解
🎯 匹配领域: 支柱九：具身大模型 (Embodied Foundation Models)
关键词: 大语言模型 偏好引导 激励机制 行为研究 安全性 模型训练 输出质量
📋 核心要点
- 现有研究表明，LLMs在偏好引导中展现出一致性，但这些偏好未必能有效转化为激励，影响模型行为。
- 本文设计了一种新的实验范式，通过常见写作任务探讨偏好与输出质量之间的关系，验证偏好的激励作用。
- 实验结果显示，尽管模型在选择中表现出偏好，给予高效用激励并未提升其输出质量，揭示了偏好与激励之间的差距。
📝 摘要（中文）
近期关于大语言模型（LLMs）偏好引导的研究表明，当面临一系列选择时，LLMs展现出一致的、特定于模型的效用结构。然而，这种结构常常包含训练者未预期的偏好，可能导致模型形成不一致的目标，进而影响安全性。本文设计了一种实验范式，以探讨这些偏好是否在现实场景中驱动LLMs的行为。研究结果表明，尽管模型在选择范式中展现出一致的偏好，但在实际写作任务中，提供高效用激励并未显著提升输出质量。由此可见，偏好的存在并不意味着它们对模型行为具有激励价值。
🔬 方法详解
问题定义：本文旨在解决大语言模型在偏好引导与实际行为之间的差距，现有方法未能有效验证偏好是否能转化为激励，导致模型行为不一致。
核心思路：通过设计一系列常见写作任务，评估模型在不同激励条件下的输出质量，探讨偏好是否能作为有效的行为激励。
技术框架：研究首先重现了偏好的引导过程，接着创建了包括论文、资助提案摘要、事件后评估和翻译在内的写作任务，最后通过高效用激励测试模型输出质量。
关键创新：本文的创新在于提出了一种新的实验范式，系统地评估了偏好与激励之间的关系，挑战了传统观点，表明偏好并不一定能转化为激励。
关键设计：在实验中，使用了盲评的独立LLM评审小组来评估输出质量，确保了结果的客观性和可靠性，同时设计了多种激励条件以测试模型的响应。
🖼️ 关键图片
📊 实验亮点
实验结果显示，在所有测试的任务中，提供高效用激励并未显著提升LLMs的输出质量，甚至在某些情况下，模型在未提供激励的条件下表现更佳。这一发现挑战了偏好与激励之间的传统关联，强调了对模型行为的深入理解的重要性。
🎯 应用场景
该研究对大语言模型的行为理解具有重要意义，尤其是在安全性和伦理性方面。通过揭示偏好与激励之间的差距，研究为未来模型的设计和训练提供了新的视角，可能影响模型在实际应用中的表现和决策过程。
📄 摘要（原文）
Recent work on preference elicitation in large language models (LLMs) has demonstrated that, when given a series of choices between two outcomes, LLMs reveal a coherent, model-specific utility structure. Notably, this structure often includes preferences that the models' trainers did not intend, such as valuing people of some nationalities above others, raising the possibility that LLMs might be forming emergent, misaligned goals, which, if true, would have major safety implications. However, the choice paradigms in which these preferences are observed are not reflective of real-world situations in which misaligned behavior would be a practical concern. Therefore, we design an experimental paradigm to probe whether these preferences serve as motivations for LLM behavior in realistic scenarios. First, we reproduce prior findings on consistent preference elicitation. Next, we create a set of common writing tasks - essays, grant proposal abstracts, incident postmortems, and translations - where quality can be assessed by a blind, independent LLM judge panel. Then, we demonstrate that LLMs can be motivated via direct exhortation and other explicit cues to modulate their output quality on these tasks. Finally, we probe whether utilities inferred from explicitly reported preferences can shift output quality on these tasks by offering LLMs high-utility incentives for high-quality outputs. In all tasks, across all models tested, offering LLMs outcomes that they report in the choice paradigm as being highly preferred does not lead them to create higher quality outputs than offering them dispreferred outcomes, or even no outcomes at all. We conclude that the existence of coherent preferences as demonstrated in choice paradigms should not be taken as evidence that those preferences have incentive value for the models or affect their behavior in other contexts.