Distilling Reasoning Ability from Large Language Models with Adaptive Thinking

📄 arXiv: 2404.09170v6 📥 PDF

作者: Xiaoshu Chen, Sihang Zhou, Ke Liang, Xinwang Liu

分类: cs.CL

发布日期: 2024-04-14 (更新: 2025-08-07)

备注: This work has been accepted by IEEE for publication. Early access in IEEE Transactions on Neural Networks and Learning Systems

💡 一句话要点

提出自适应思维机制以提升小型语言模型的推理能力

🎯 匹配领域: 支柱九：具身大模型 (Embodied Foundation Models)

关键词: 小型语言模型 推理能力 链式思维微调 后思维机制 自适应思维 软提示微调 复杂问题解决

📋 核心要点

  1. 现有的链式思维微调方法在推理过程中对小错误极为敏感，影响答案的正确性。
  2. 本文提出了一种后思维机制，允许模型先生成答案，再生成推理过程，从而提高鲁棒性和推理效率。
  3. 在12个推理任务和2个代表性语言模型上进行的实验表明，提出的自适应思维机制显著提升了SLM的性能。

📝 摘要（中文）

链式思维微调（cot-finetuning）旨在通过模仿大型语言模型（LLM）的推理过程，使小型语言模型（SLM）具备推理能力，从而提升其在特定任务上的表现。现有方法多采用预思维机制，允许SLM在给出答案前生成推理过程，这虽然能帮助SLM分析复杂问题，但也使得答案的正确性对推理过程中的小错误高度敏感。为此，本文提出了一种稳健的后思维机制，允许SLM先生成答案，再生成推理过程。该机制不仅能避免小错误对答案的负面影响，还能使推理效率提高。尽管后思维机制带来了诸多优势，但可能会损失对问题的思考能力。为此，本文提出了一种插拔式自适应思维机制，通过软提示微调，结合前后思维机制的优点，动态决定SLM是先回答还是先思考。大量实验验证了该机制的有效性。

🔬 方法详解

问题定义：本文解决的是小型语言模型在推理任务中对小错误敏感的问题，现有的预思维机制导致答案的正确性受到影响。

核心思路：提出了一种后思维机制，允许模型先生成答案，再生成推理过程，从而避免小错误对答案的负面影响，并提高推理效率。

技术框架：整体架构包括一个感知模块，该模块根据问题的复杂性动态决定SLM是先回答还是先思考。通过软提示微调，整合前后思维机制的优点。

关键创新：最重要的创新在于提出了后思维机制，并结合自适应思维机制，使SLM能够根据问题复杂性灵活选择推理策略，与现有方法相比，显著提升了模型的鲁棒性和效率。

关键设计：在模型设计中，采用了软提示微调技术，设置了特定的损失函数以优化推理过程，并设计了感知模块以评估问题复杂性。

🖼️ 关键图片

fig_0
fig_1
fig_2

📊 实验亮点

实验结果显示，采用自适应思维机制的SLM在12个推理任务上相较于基线模型性能提升了显著，具体提升幅度达到XX%，验证了该机制的有效性和实用性。

🎯 应用场景

该研究的潜在应用领域包括教育、智能问答系统和复杂问题解决等场景。通过提升小型语言模型的推理能力，可以在资源受限的环境中实现更高效的智能交互，具有重要的实际价值和未来影响。

📄 摘要（原文）

Chain of thought finetuning (cot-finetuning) aims to endow small language models (SLM) with reasoning ability to improve their performance towards specific tasks by allowing them to imitate the reasoning procedure of large language models (LLM) beyond simply predicting the answers. Most existing cot-finetuning methods adopt a pre-thinking mechanism, allowing the SLM to generate a rationale before providing an answer. This mechanism enables SLM to analyze and think about complex questions, but it also makes answer correctness highly sensitive to minor errors in rationale. Therefore, we propose a robust post-thinking mechanism to generate answers before rationale. Thanks to this answer-first setting, 1) the answer can escape from the adverse effects caused by minor errors in the rationale; 2) the rationale serves as an error amplifier to the answer, which makes the SLM focus on learning hard samples; 3) the inferring efficiency can also benefit from the setting since users can stop the generation right after answers are outputted when inference is conducted. However, although the post-thinking mechanism brings many advantages and improves the overall performance of SLM on specific tasks, it may lose the ability to think about the questions and decompose complex questions into simple sub-questions compared to pre-thinking mechanism. Therefore, a plug-and-play adaptive-thinking mechanism is proposed with the aid of the soft prompt tuning to integrate the merits of the pre-thinking mechanism and post-thinking mechanism, in which a perception module is introduced to adaptively prompt SLM answer or think first based on perceiving the complexity of the questions. Extensive experiments are conducted across 12 reasoning tasks and 2 representative language models to demonstrate the effectiveness of the proposed mechanism.