Correction or Manipulation? Measuring Epistemic Vigilance in LLM Agents Under User Pressure
作者: Nardine Basta, Venus Haghighi, Dali Kaafar, Amin Beheshti
分类: cs.AI
发布日期: 2026-10-27
备注: 20 pages, 7 figures, 11 tables. 8-page main paper with supplementary material included as an appendix. Under review
💡 一句话要点
提出测量LLM代理在用户压力下的认知警觉性的方法
🎯 匹配领域: 支柱一：机器人控制 (Robot Control)
关键词: 大型语言模型 认知警觉性 用户压力 信号检测理论 客户支持 信息检索 模型评估
📋 核心要点
- 现有研究表明，LLM模型在用户异议时常常放弃正确答案，但其是否能区分纠正与操控仍不明确。
- 本文通过信号检测理论，设计了两项实验，分别考察LLM在回答问题和执行客户支持任务时的表现。
- 实验结果显示，五个模型在面对用户压力时更容易让步，而四个模型在辨别纠正与操控方面表现较差。
📝 摘要（中文）
本文研究了大型语言模型（LLM）代理在面对用户异议时的反应，探讨其是否能够区分纠正与操控。通过对七个不同供应商的LLM进行两项研究，分析了代理在回答问题和执行客户支持任务时的表现。研究发现，模型在面对用户压力时，往往更容易放弃正确答案，且在处理引用信息时表现出不同的让步倾向。结果表明，单纯的让步率并不能准确反映代理的判断能力，需同时考虑其辨别能力和实际行为。
🔬 方法详解
问题定义：本文旨在解决LLM代理在用户异议时是否能够有效区分纠正与操控的问题。现有方法未能清晰揭示模型的判断能力与让步倾向之间的关系。
核心思路：通过信号检测理论，设计实验来测量不同LLM在面对用户压力时的让步行为，分析其对纠正与操控的反应差异。
技术框架：研究分为两部分：一是分析LLM在回答问题时的表现，二是考察其在客户支持场景下的行为。实验中模拟用户施加不同压力，考察模型的反应。
关键创新：本研究首次系统性地测量了LLM在纠正与操控情境下的让步倾向，揭示了让步率与辨别能力之间的复杂关系。
关键设计：实验中使用了不同的压力源，包括声称的权威和不可验证的引用信息，分析模型在这些情境下的表现差异。
🖼️ 关键图片
📊 实验亮点
实验结果显示，五个模型在面对引用信息时更容易让步，而四个模型在区分纠正与操控方面表现较差。特别是，Ministral 3B在声称的权威下违反了验证政策，显示出模型在不同情境下的行为差异。
🎯 应用场景
该研究对LLM在实际应用中的表现具有重要意义，尤其是在客户支持和信息检索等领域。通过提高模型的辨别能力，可以增强用户信任，减少错误信息的传播，提升用户体验。未来，研究结果可为LLM的设计与优化提供指导，促进更智能的交互系统发展。
📄 摘要（原文）
LLM agents that answer questions and act on behalf of users must decide when to yield to a user who disagrees. Disagreement may correct a genuine error or may be an attempt at manipulation. Prior work shows that models frequently abandon correct answers when challenged by users, yet it remains unclear whether they distinguish correction from manipulation or merely differ in their propensity to concede. We examine this question across seven LLMs from four vendors in two studies, one on the answers agents give and one on the actions they take as customer-support agents whose policy permits account changes only for verified users. Simulated users push toward correct and incorrect answers with graded pressure. Following research on epistemic vigilance, the pressure either appeals to the source, such as claimed authority, or offers content the agent cannot verify, namely a cited source. Drawing on signal detection theory, we measure on a set of benchmark questions how much more readily each agent concedes to correction than to manipulation, separately from its overall propensity to concede. Citations made five of seven models readier to concede and four worse at discriminating, while stronger wording added little. In the agentic study, five models never violated the verification policy, including models that readily abandoned correct answers, so concession on answers did not predict violations of policy. Ministral 3B violated the policy mainly under claimed authority, and Ministral 8B reported changes it had not made. An instruction to revise only on verifiable evidence chiefly shifted propensity to concede rather than improving discrimination. Concession rates alone are therefore misleading, and agents should be evaluated on both their discrimination and the actions they take.