The Riddle Riddle: Testing Flexible Reasoning in Large Language Models and Humans

📄 arXiv: 2606.27103v1 📥 PDF

作者: Bella Fascendini, Kathryn McGregor, Max D. Gupta, Thomas L. Griffiths

分类: cs.CL

发布日期: 2026-06-25

💡 一句话要点

提出谜题谜题范式以测试大型语言模型的灵活推理能力

🎯 匹配领域: 支柱九：具身大模型 (Embodied Foundation Models)

关键词: 大型语言模型 灵活推理 认知任务 谜题谜题 实验研究 推理策略 人类与模型对比

📋 核心要点

  1. 现有大型语言模型在认知任务中的表现令人瞩目，但其推理能力的灵活性仍然存在疑问。
  2. 本文提出谜题谜题范式，旨在通过设计特定的文字问题来测试LLMs和人类的推理策略灵活性。
  3. 实验结果显示，LLMs在真实谜题上的准确率为84.9%，而在谜题谜题上的准确率仅为50.7%，反之人类表现则相反。

📝 摘要（中文）

人类能够灵活调整推理策略以应对不同问题，而大型语言模型（LLMs）在许多认知任务中表现良好，但其准确性是否源于训练数据的模式匹配或灵活推理尚不明确。本文提出了一种新颖的测试范式——谜题谜题范式，旨在探讨这一问题。通过两个实验，研究发现LLMs在真实谜题上的表现显著优于在谜题谜题上的表现，而人类则表现相反。这表明LLMs的强大表现可能反映了记忆检索而非灵活策略选择。

🔬 方法详解

问题定义：本文旨在解决大型语言模型在推理任务中的灵活性问题，现有方法未能有效区分模型的模式匹配与灵活推理能力。

核心思路：通过设计谜题谜题，要求模型在字面解释与灵活推理之间进行选择，以测试其推理策略的适应性。

技术框架：研究分为两个实验，涉及九种最先进的LLMs和100名人类参与者，比较其在真实谜题与谜题谜题上的表现。

关键创新：引入谜题谜题范式作为评估LLMs推理能力的工具，揭示了模型在不同推理策略下的表现差异。

关键设计：实验中对谜题谜题的设计进行了细致考量，确保其答案仅需字面解释，同时通过错误分析揭示了LLMs与人类在推理错误上的不同来源。

🖼️ 关键图片

fig_0
fig_1
fig_2

📊 实验亮点

实验结果显示，LLMs在真实谜题上的准确率为84.9%，而在谜题谜题上的准确率仅为50.7%。相对而言，人类在真实谜题上的表现为50.5%，而在谜题谜题上的表现为80.5%。这一对比揭示了LLMs在推理策略上的局限性。

🎯 应用场景

该研究为理解大型语言模型的推理能力提供了新的视角，潜在应用于教育、心理学和人工智能领域，帮助开发更具灵活性的智能系统。未来可能影响模型设计和评估标准，推动更高效的推理算法的研究与应用。

📄 摘要（原文）

Humans flexibly adapt their reasoning strategies to the requirements of a given problem. Large language models (LLMs) have performed well on many cognitive tasks, however, it is unclear whether this accuracy is a result of pattern matching from training data or flexible reasoning. Here, we introduce a novel paradigm to test this question: the riddle riddle paradigm. Riddle riddles are word problems written to mimic popular riddles, but altered so their answers only require literal interpretations. Identifying correct answers requires looking past the structure of each question and flexibly apply different reasoning strategies based on the content. If LLMs respond to surface features, such as form, a riddle-like structure should cause models to use an inventive reasoning strategy even when a literal interpretation suffices. Alternatively, if LLMs reason based on content, they should flexibly switch strategies when appropriate. Across two experiments with nine state-of-the-art LLMs and 100 human participants, we show humans and LLMs fail on this paradigm in opposite directions. LLMs were far more accurate on genuine riddles than on riddle riddles (84.9% vs. 50.7%); whereas humans showed the reverse effect (50.5% vs. 80.5%). Error analysis shows that 90.8% of LLM errors on riddle riddles (the condition where they show diminished performance) were due to inappropriate use of inventive reasoning while only 57.6% of human errors on genuine riddles were due to overextending literal reasoning. Thus, while both groups make mistakes, reasoning mistakes are made more often by LLMs than by humans. Overall, LLMs' strong performance on genuine riddles may reflect memory retrieval rather than flexible strategy selection, and without stimuli designed to elicit this contrast, it becomes easy to conflate LLM-generated outputs that look like reasoning with genuine reasoning.