Probabilistic Tree-of-thought Reasoning for Answering Knowledge-intensive Complex Questions

📄 arXiv: 2311.13982v1 📥 PDF

作者: Shulin Cao, Jiajie Zhang, Jiaxin Shi, Xin Lv, Zijun Yao, Qi Tian, Juanzi Li, Lei Hou

分类: cs.CL, cs.AI

发布日期: 2023-11-23

备注: Accepted by EMNLP 2023

💡 一句话要点

提出概率树思维推理以解决知识密集型复杂问题

🎯 匹配领域: 支柱九：具身大模型 (Embodied Foundation Models)

关键词: 知识密集型问答 概率推理 链式思维 查询树 大型语言模型 信息检索 智能问答系统

📋 核心要点

  1. 现有方法在知识密集型复杂问题的推理中存在负面检索和局部错误传播的问题。
  2. 本文提出的概率树思维推理通过构建查询树并自底向上进行概率推理，提升了推理的准确性和可靠性。
  3. 实验结果显示，ProbTree在多个复杂问答数据集上显著优于当前最先进的方法，验证了其有效性。

📝 摘要（中文）

大型语言模型（LLMs）能够通过链式思维（CoT）推理回答知识密集型复杂问题。然而，当模型参数中的知识不可用或过时时，它们往往会生成事实错误的推理步骤。近期研究开始利用外部知识来增强CoT推理，但链式方法存在负面检索和视野有限的问题。本文提出了一种新方法：概率树思维推理（ProbTree），通过将复杂问题转化为查询树，进行自底向上的概率推理，从而克服了这些问题。实验结果表明，该方法在三个复杂问答数据集上显著优于现有最先进的方法。

🔬 方法详解

问题定义：本文旨在解决大型语言模型在回答知识密集型复杂问题时，因缺乏必要知识而导致的推理错误，尤其是负面检索和局部错误传播的问题。

核心思路：论文提出通过构建查询树来分解复杂问题，并采用概率推理的方法，从叶节点到根节点逐步解决子问题，增强推理的全局视野和准确性。

技术框架：整体架构包括将复杂问题转化为查询树，树的每个非根节点表示其父节点的子问题。推理过程从叶节点开始，结合闭卷问答和开卷问答的答案选择，逐步向上推理。

关键创新：最重要的创新在于引入概率推理机制，使得模型在选择答案时能够考虑到问题分解和回答的置信度，从而有效避免负面检索和局部错误的影响。

关键设计：在设计中，模型通过选择更有信心的答案（来自闭卷或开卷问答），并利用层次结构进行全局推理，确保信息的有效整合与传播。

🖼️ 关键图片

fig_0
fig_1
fig_2

📊 实验亮点

实验结果表明，ProbTree在三个复杂问答数据集上相较于现有最先进的方法有显著提升，具体表现为在准确率上提高了XX%，有效验证了概率树思维推理的有效性。

🎯 应用场景

该研究的潜在应用领域包括智能问答系统、教育辅助工具和信息检索等。通过提高复杂问题的回答准确性，ProbTree能够在实际应用中提供更可靠的知识支持，推动智能系统的进一步发展。

📄 摘要（原文）

Large language models (LLMs) are capable of answering knowledge-intensive complex questions with chain-of-thought (CoT) reasoning. However, they tend to generate factually incorrect reasoning steps when the required knowledge is not available or up-to-date in models' parameters. Recent works turn to retrieving external knowledge to augment CoT reasoning. Despite being promising, these chain-based methods suffer from: 1) Negative retrieval. Unnecessary or incorrect retrieval may mislead the reasoning; 2) Limited sight. Lacking the ability to look backward or forward, a local error in one step will propagate along the chain. In this paper, we propose a novel approach: Probabilistic Tree-of-thought Reasoning (ProbTree). First, LLMs translate a complex question into a query tree, in which each non-root node denotes a sub-question of its parent node. Then, probabilistic reasoning is conducted over the tree, by solving questions from leaf to root considering the confidence of both question decomposing and answering. During reasoning, for leaf nodes, LLMs choose a more confident answer from Closed-book QA that employs parametric knowledge and Open-book QA that employs retrieved external knowledge, thus eliminating the negative retrieval problem. For non-leaf nodes, with the hierarchical structure, LLMs have broader sights and are able to globally reason with the information from child nodes, thus recovering from local errors. The experiments on three Complex QA datasets under the open-domain setting show that our approach outperforms SOTA methods significantly, demonstrating the effect of probabilistic tree-of-thought reasoning.