The effect of source disclosure on evaluation of AI-generated messages: A two-part study
作者: Sue Lim, Ralf Schmälzle
分类: cs.CL
发布日期: 2023-11-27 (更新: 2023-11-28)
备注: Manuscript currently under review. Paper presented at 109th Annual National Communication Association (NCA) Conference, November 16-19, 2023. 10 pages, 5 figures. Supplementary file formatting updated in current version
DOI: 10.1016/j.chbah.2024.100058
💡 一句话要点
研究源披露对AI生成消息评估的影响
🎯 匹配领域: 支柱九：具身大模型 (Embodied Foundation Models)
关键词: AI生成内容 源披露 健康传播 消息评估 负面态度 公共健康 沟通行为
📋 核心要点
- 核心问题：当前对AI生成消息的评估缺乏对源披露影响的系统研究，尤其是在健康传播领域。
- 方法要点：通过两项研究，探讨源披露对AI与人类生成消息评估的影响，并分析负面态度的调节作用。
- 实验或效果：研究发现源披露显著影响消息评估，且对AI持有负面态度的参与者对AI生成消息的偏好降低。
📝 摘要（中文）
近年来，人工智能（AI）的进步表明机器能够展现沟通行为，并影响人类的思维、情感和行为。本文探讨了消息源披露如何影响人们对AI生成消息与人类生成消息的评估。在一项预注册的研究中，我们发现源披露显著影响了人们对AI生成健康预防消息的评估，但未显著改变消息排名。在后续研究中，我们发现参与者对AI的负面态度在消息评估中起到显著的调节作用。总体而言，研究结果显示在源披露后，存在对AI生成消息的轻微偏见，丰富了AI与沟通交叉领域的研究。
🔬 方法详解
问题定义：本文旨在解决源披露对AI生成消息评估的影响问题。现有研究未充分探讨源披露如何影响人们对AI生成内容的接受度和偏好，尤其是在健康传播的背景下。
核心思路：研究通过比较AI与人类生成消息的评估，揭示源披露对消息接受度的影响，并分析参与者对AI的态度如何调节这一影响。设计旨在深入理解人类对AI生成内容的认知偏见。
技术框架：研究分为两部分，第一部分通过实验设计评估源披露的影响，第二部分分析参与者的负面态度如何影响消息评估。主要模块包括消息生成、源披露、评估指标等。
关键创新：本研究的创新点在于系统性地探讨源披露对AI生成消息的评估影响，填补了AI与人类生成内容比较研究的空白，尤其是在健康传播领域的应用。
关键设计：研究采用预注册设计，确保结果的可靠性。评估指标包括消息的可信度、吸引力和偏好度，参与者的负面态度通过问卷量表进行测量。
🖼️ 关键图片
📊 实验亮点
实验结果显示，源披露显著影响了参与者对AI生成健康预防消息的评估，尤其是对持有负面态度的参与者，偏好度降低。具体而言，源披露后，参与者对AI生成消息的偏见略有增加，表明源信息在消息接受度中的重要性。
🎯 应用场景
该研究的潜在应用场景包括公共健康传播、教育和市场营销等领域。通过理解源披露对AI生成内容的影响，可以优化AI工具在这些领域的应用，提高信息传播的有效性和接受度。未来，随着AI技术的普及，研究结果将对如何设计和使用AI生成内容提供重要指导。
📄 摘要（原文）
Advancements in artificial intelligence (AI) over the last decade demonstrate that machines can exhibit communicative behavior and influence how humans think, feel, and behave. In fact, the recent development of ChatGPT has shown that large language models (LLMs) can be leveraged to generate high-quality communication content at scale and across domains, suggesting that they will be increasingly used in practice. However, many questions remain about how knowing the source of the messages influences recipients' evaluation of and preference for AI-generated messages compared to human-generated messages. This paper investigated this topic in the context of vaping prevention messaging. In Study 1, which was pre-registered, we examined the influence of source disclosure on people's evaluation of AI-generated health prevention messages compared to human-generated messages. We found that source disclosure (i.e., labeling the source of a message as AI vs. human) significantly impacted the evaluation of the messages but did not significantly alter message rankings. In a follow-up study (Study 2), we examined how the influence of source disclosure may vary by the participants' negative attitudes towards AI. We found a significant moderating effect of negative attitudes towards AI on message evaluation, but not for message selection. However, for those with moderate levels of negative attitudes towards AI, source disclosure decreased the preference for AI-generated messages. Overall, the results of this series of studies showed a slight bias against AI-generated messages once the source was disclosed, adding to the emerging area of study that lies at the intersection of AI and communication.