Accuracy of a Large Language Model in Distinguishing Anti- And Pro-vaccination Messages on Social Media: The Case of Human Papillomavirus Vaccination

📄 arXiv: 2404.06731v1 📥 PDF

作者: Soojong Kim, Kwanho Kim, Claire Wonjeong Jo

分类: cs.CY, cs.AI

发布日期: 2024-04-10

备注: Forthcoming in Preventive Medicine Reports

💡 一句话要点

利用大型语言模型分析社交媒体对HPV疫苗的态度

🎯 匹配领域: 支柱九：具身大模型 (Embodied Foundation Models)

关键词: 大型语言模型 社交媒体分析 疫苗态度 情感分析 公共健康

📋 核心要点

  1. 社交媒体上关于疫苗的公众意见存在分歧，现有分析方法难以高效处理这些信息。
  2. 本研究利用ChatGPT对HPV疫苗相关社交媒体信息进行情感分析，评估其分类准确性。
  3. 实验结果表明，使用多个响应实例可以显著提高分类准确率，尤其是在长格式信息中。

📝 摘要（中文）

本研究旨在评估大型语言模型（LLM）ChatGPT在分析社交媒体上关于人乳头瘤病毒（HPV）疫苗的支持与反对信息的准确性。研究收集了来自Facebook和Twitter的相关信息，并对1,000条经过人工评估的消息进行了分类。结果显示，使用20个响应实例时，模型在长格式反对和支持信息的分类准确率分别为0.882和0.750，而短格式的准确率为0.773和0.723。尽管模型在分类反对信息时表现更佳，但在公共健康背景下理解其局限性仍然至关重要。

🔬 方法详解

问题定义：本研究旨在解决社交媒体上关于HPV疫苗的支持与反对信息分类的准确性问题。现有方法在处理不同格式的社交媒体信息时，准确性和效率存在不足。

核心思路：本研究利用ChatGPT这一大型语言模型，通过分析社交媒体信息的情感倾向，来识别公众对HPV疫苗的态度。选择多次响应实例以提高分类的准确性。

技术框架：研究首先从Facebook和Twitter收集HPV疫苗相关消息，随后对1,000条消息进行人工评估，并输入LLM生成分类结果，最后通过比较人类与机器的分类结果来评估准确性。

关键创新：本研究的创新点在于将大型语言模型应用于公共健康领域的社交媒体信息分析，尤其是在疫苗相关的情感分析上，展示了其在分类任务中的潜力。

关键设计：在实验中，使用了不同数量的响应实例（如1、3和20个），并测量了每条消息的分类准确性，发现使用20个实例时准确性显著提高。

📊 实验亮点

实验结果显示，使用20个响应实例时，ChatGPT在长格式反对信息的分类准确率达到0.882，支持信息的准确率为0.750，短格式信息的准确率分别为0.773和0.723，表明模型在处理社交媒体信息时具有较高的准确性。

🎯 应用场景

该研究为公共健康领域提供了一种新的工具，通过大型语言模型分析社交媒体上的公众意见，能够帮助政策制定者和健康传播者更好地理解和应对疫苗接种的公众态度，具有重要的实际价值和应用潜力。

📄 摘要（原文）

Objective. Vaccination has engendered a spectrum of public opinions, with social media acting as a crucial platform for health-related discussions. The emergence of artificial intelligence technologies, such as large language models (LLMs), offers a novel opportunity to efficiently investigate public discourses. This research assesses the accuracy of ChatGPT, a widely used and freely available service built upon an LLM, for sentiment analysis to discern different stances toward Human Papillomavirus (HPV) vaccination. Methods. Messages related to HPV vaccination were collected from social media supporting different message formats: Facebook (long format) and Twitter (short format). A selection of 1,000 human-evaluated messages was input into the LLM, which generated multiple response instances containing its classification results. Accuracy was measured for each message as the level of concurrence between human and machine decisions, ranging between 0 and 1. Results. Average accuracy was notably high when 20 response instances were used to determine the machine decision of each message: .882 (SE = .021) and .750 (SE = .029) for anti- and pro-vaccination long-form; .773 (SE = .027) and .723 (SE = .029) for anti- and pro-vaccination short-form, respectively. Using only three or even one instance did not lead to a severe decrease in accuracy. However, for long-form messages, the language model exhibited significantly lower accuracy in categorizing pro-vaccination messages than anti-vaccination ones. Conclusions. ChatGPT shows potential in analyzing public opinions on HPV vaccination using social media content. However, understanding the characteristics and limitations of a language model within specific public health contexts remains imperative.