AI-Augmented Predictions: LLM Assistants Improve Human Forecasting Accuracy

📄 arXiv: 2402.07862v2 📥 PDF

作者: Philipp Schoenegger, Peter S. Park, Ezra Karger, Sean Trott, Philip E. Tetlock

分类: cs.CY, cs.AI, cs.CL, cs.LG

发布日期: 2024-02-12 (更新: 2024-08-22)

备注: 22 pages pages (main text comprised of 19 pages, appendix comprised of three pages). 10 visualizations in the main text (four figures, six tables), three additional figures in the appendix

💡 一句话要点

利用LLM助手提升人类预测准确性

🎯 匹配领域: 支柱九：具身大模型 (Embodied Foundation Models)

关键词: 大型语言模型 预测准确性 决策支持 超预测 噪声预测 人类判断 认知偏差

📋 核心要点

  1. 现有预测方法在准确性和可靠性上存在不足，尤其是在复杂任务中人类判断容易受到认知偏差的影响。
  2. 本研究提出利用两种不同类型的LLM助手来增强人类预测能力，分别为高质量建议和噪声预测助手。
  3. 实验结果表明，使用LLM助手的参与者预测准确性显著提高，超预测助手的提升幅度更为明显，显示出LLM在决策支持中的潜力。

📝 摘要（中文）

大型语言模型（LLMs）在多个领域的表现与人类相匹配，甚至超越。本研究探讨了LLMs在预测任务中增强人类判断的潜力。我们评估了两种LLM助手对人类预测者的影响：一种提供高质量建议，另一种则提供噪声预测。参与者（N = 991）回答了六个预测问题，并可以随时咨询分配的LLM助手。结果显示，与控制组相比，使用LLM助手的参与者预测准确性显著提高了24%至28%。进一步分析表明，超预测助手的准确性提升达到41%，而噪声助手为29%。研究结果表明，LLM助手在认知要求高的任务中可作为有效的决策辅助工具。

🔬 方法详解

问题定义：本研究旨在解决人类在复杂预测任务中的判断偏差和准确性不足的问题。现有方法往往缺乏有效的决策支持，导致预测结果不理想。

核心思路：通过引入两种类型的LLM助手，分别提供高质量和噪声预测建议，以增强人类预测者的判断能力。这样的设计旨在利用LLM的强大语言理解和生成能力，帮助人类克服认知偏差。

技术框架：研究中参与者被分为三组：使用高质量LLM助手、使用噪声LLM助手和控制组。参与者回答六个预测问题，并可以随时咨询其分配的助手。

关键创新：本研究的主要创新在于比较不同类型的LLM助手对人类预测准确性的影响，尤其是超预测助手与噪声助手的效果差异，这为LLM在决策支持中的应用提供了新的视角。

关键设计：在实验中，LLM助手的设计包括高质量建议的生成机制和噪声预测的故意设计。参与者的反馈和预测结果被用作评估助手效果的关键指标。

🖼️ 关键图片

fig_0
fig_1
fig_2

📊 实验亮点

实验结果显示，使用LLM助手的参与者预测准确性提高了24%至28%。超预测助手的准确性提升达到41%，而噪声助手为29%。这些结果表明，LLM助手在认知要求高的任务中具有显著的决策辅助效果。

🎯 应用场景

该研究的潜在应用领域包括商业决策、金融市场预测、政策制定等需要高准确性判断的场景。通过利用LLM助手，决策者可以在复杂和不确定的环境中获得更可靠的预测支持，提升决策质量。未来，随着LLM技术的进一步发展，其在各行业的应用价值将更加显著。

📄 摘要（原文）

Large language models (LLMs) match and sometimes exceeding human performance in many domains. This study explores the potential of LLMs to augment human judgement in a forecasting task. We evaluate the effect on human forecasters of two LLM assistants: one designed to provide high-quality ("superforecasting") advice, and the other designed to be overconfident and base-rate neglecting, thus providing noisy forecasting advice. We compare participants using these assistants to a control group that received a less advanced model that did not provide numerical predictions or engaged in explicit discussion of predictions. Participants (N = 991) answered a set of six forecasting questions and had the option to consult their assigned LLM assistant throughout. Our preregistered analyses show that interacting with each of our frontier LLM assistants significantly enhances prediction accuracy by between 24 percent and 28 percent compared to the control group. Exploratory analyses showed a pronounced outlier effect in one forecasting item, without which we find that the superforecasting assistant increased accuracy by 41 percent, compared with 29 percent for the noisy assistant. We further examine whether LLM forecasting augmentation disproportionately benefits less skilled forecasters, degrades the wisdom-of-the-crowd by reducing prediction diversity, or varies in effectiveness with question difficulty. Our data do not consistently support these hypotheses. Our results suggest that access to a frontier LLM assistant, even a noisy one, can be a helpful decision aid in cognitively demanding tasks compared to a less powerful model that does not provide specific forecasting advice. However, the effects of outliers suggest that further research into the robustness of this pattern is needed.