The Promise and Challenges of Using LLMs to Accelerate the Screening Process of Systematic Reviews

📄 arXiv: 2404.15667v4 📥 PDF

作者: Aleksi Huotala, Miikka Kuutila, Paul Ralph, Mika Mäntylä

分类: cs.CL, cs.AI

发布日期: 2024-04-24 (更新: 2024-05-08)

备注: Accepted to the International Conference on Evaluation and Assessment in Software Engineering (EASE), 2024 edition

💡 一句话要点

利用大型语言模型加速系统评价筛选过程的研究

🎯 匹配领域: 支柱九：具身大模型 (Embodied Foundation Models)

关键词: 大型语言模型 系统评价 文本简化 自动化筛选 软件工程 提示技术 人类筛选者

📋 核心要点

  1. 系统评价的实施周期长达67周，现有方法在效率上存在显著不足。
  2. 本文通过使用大型语言模型（LLMs）来简化摘要和自动化标题-摘要筛选，旨在提高筛选效率。
  3. 实验结果显示，GPT-4在筛选任务中表现优于GPT-3.5，Few-shot和One-shot提示法的效果优于Zero-shot，但LLMs的准确性仍需提升。

📝 摘要（中文）

系统评价（SR）是软件工程领域常用的研究方法，但其实施平均需要67周。本文探讨了大型语言模型（LLMs）在简化摘要和自动化标题-摘要筛选中的应用。通过实验，比较了人类筛选者与GPT-3.5和GPT-4在筛选任务中的表现。研究发现，文本简化未显著提高筛选效果，但减少了筛选时间。GPT-4在性能上优于GPT-3.5，Few-shot和One-shot提示法表现优于Zero-shot。尽管LLMs在自动化筛选中展现出潜力，但其准确性尚未显著超越人类筛选者。未来需要更多研究来验证LLMs在SR筛选过程中的有效性。

🔬 方法详解

问题定义：本文旨在解决系统评价（SR）筛选过程中的时间效率问题，现有方法耗时较长，影响研究进度。

核心思路：通过引入大型语言模型（LLMs），简化文本摘要并自动化标题-摘要筛选，以期减少人类筛选者的工作量和时间。

技术框架：研究分为几个主要模块：首先是人类筛选者对原始和简化摘要的筛选实验；其次是使用GPT-3.5和GPT-4进行相同筛选任务；最后，比较不同提示技术（Zero-shot、One-shot、Few-shot及Few-shot with Chain-of-Thought）对筛选性能的影响。

关键创新：本文的创新在于将LLMs应用于系统评价的筛选过程，尤其是通过不同提示技术的比较，揭示了LLMs在文本简化和自动化筛选中的潜力与局限。

关键设计：实验中对提示的设计进行了优化，采用了多种提示技术，结果表明Few-shot和One-shot提示法在性能上优于Zero-shot，且GPT-4的表现优于GPT-3.5。

📊 实验亮点

实验结果显示，GPT-4在筛选任务中的表现优于GPT-3.5，Few-shot和One-shot提示法的效果显著优于Zero-shot提示。尽管LLMs在自动化筛选中展现出潜力，但其准确性尚未显著超越人类筛选者，表明仍需进一步研究。

🎯 应用场景

该研究为系统评价的筛选过程提供了新的思路，尤其是在软件工程领域。通过使用大型语言模型，研究人员可以在一定程度上提高筛选效率，减少人力成本。未来，随着LLMs技术的进步，其在科研中的应用潜力将进一步扩大。

📄 摘要（原文）

Systematic review (SR) is a popular research method in software engineering (SE). However, conducting an SR takes an average of 67 weeks. Thus, automating any step of the SR process could reduce the effort associated with SRs. Our objective is to investigate if Large Language Models (LLMs) can accelerate title-abstract screening by simplifying abstracts for human screeners, and automating title-abstract screening. We performed an experiment where humans screened titles and abstracts for 20 papers with both original and simplified abstracts from a prior SR. The experiment with human screeners was reproduced with GPT-3.5 and GPT-4 LLMs to perform the same screening tasks. We also studied if different prompting techniques (Zero-shot (ZS), One-shot (OS), Few-shot (FS), and Few-shot with Chain-of-Thought (FS-CoT)) improve the screening performance of LLMs. Lastly, we studied if redesigning the prompt used in the LLM reproduction of screening leads to improved performance. Text simplification did not increase the screeners' screening performance, but reduced the time used in screening. Screeners' scientific literacy skills and researcher status predict screening performance. Some LLM and prompt combinations perform as well as human screeners in the screening tasks. Our results indicate that the GPT-4 LLM is better than its predecessor, GPT-3.5. Additionally, Few-shot and One-shot prompting outperforms Zero-shot prompting. Using LLMs for text simplification in the screening process does not significantly improve human performance. Using LLMs to automate title-abstract screening seems promising, but current LLMs are not significantly more accurate than human screeners. To recommend the use of LLMs in the screening process of SRs, more research is needed. We recommend future SR studies publish replication packages with screening data to enable more conclusive experimenting with LLM screening.