Mathify: Evaluating Large Language Models on Mathematical Problem Solving Tasks

📄 arXiv: 2404.13099v1 📥 PDF

作者: Avinash Anand, Mohit Gupta, Kritarth Prasad, Navya Singla, Sanjana Sanjeev, Jatin Kumar, Adarsh Raj Shivam, Rajiv Ratn Shah

分类: cs.CL, cs.AI

发布日期: 2024-04-19

备注: 10 pages, 3 figures, NeurIPS 2023 Workshop on Generative AI for Education (GAIED)

期刊: NeurIPS 2023 Workshop on Generative AI for Education (GAIED)

💡 一句话要点

提出MathQuest数据集以评估大语言模型在数学问题求解中的表现

🎯 匹配领域: 支柱九：具身大模型 (Embodied Foundation Models)

关键词: 大语言模型 数学问题求解 数据集构建 微调实验 教育技术

📋 核心要点

  1. 现有方法在评估大语言模型的算术能力方面存在不足，缺乏系统性的数据集和评估标准。
  2. 论文提出了MathQuest数据集，涵盖多种复杂度的数学问题，并对三种大语言模型进行了微调实验。
  3. 实验结果表明，MAmmoTH-13B在数学问题求解中表现最佳，成为新的性能基准。

📝 摘要（中文）

随着自然语言处理系统和大语言模型的快速发展，教育和教学方法领域出现了许多机会。这些进展为个性化学习体验和即时反馈提供了可能，尤其在数学问题求解方面。然而，大语言模型的算术能力评估仍然受到忽视。为此，本文引入了名为“MathQuest”的数学数据集，涵盖了来自11和12年级NCERT数学教科书的多样化数学挑战。通过对LLaMA-2、WizardMath和MAmmoTH三种主流大语言模型进行微调实验，结果显示MAmmoTH-13B在解决数学问题方面表现最佳，成为应对NCERT数学问题的可靠基准。

🔬 方法详解

问题定义：本文旨在评估大语言模型在数学问题求解中的算术能力，现有方法缺乏针对性的评估数据集，导致模型性能难以量化。

核心思路：通过构建MathQuest数据集，涵盖多样化的数学问题，进行大语言模型的微调，以便更好地评估其在数学求解任务中的表现。

技术框架：整体流程包括数据集构建、模型微调和性能评估三个主要阶段。数据集从NCERT教科书中提取，模型微调使用LLaMA-2、WizardMath和MAmmoTH。

关键创新：引入了MathQuest数据集，系统性地评估大语言模型的算术能力，填补了现有研究的空白。与传统方法相比，提供了更为全面的评估标准。

关键设计：在微调过程中，采用了特定的损失函数和优化策略，以确保模型能够有效学习数学问题的求解过程。

📊 实验亮点

实验结果显示，MAmmoTH-13B在解决数学问题时表现最佳，超越了LLaMA-2和WizardMath，具体性能数据未提供，但其成为NCERT数学问题的可靠基准，显示出显著的提升幅度。

🎯 应用场景

该研究的潜在应用领域包括教育技术、智能辅导系统和在线学习平台。通过提供个性化的数学问题求解支持，能够提升学生的学习效果和兴趣，未来可能对教育方式产生深远影响。

📄 摘要（原文）

The rapid progress in the field of natural language processing (NLP) systems and the expansion of large language models (LLMs) have opened up numerous opportunities in the field of education and instructional methods. These advancements offer the potential for tailored learning experiences and immediate feedback, all delivered through accessible and cost-effective services. One notable application area for this technological advancement is in the realm of solving mathematical problems. Mathematical problem-solving not only requires the ability to decipher complex problem statements but also the skill to perform precise arithmetic calculations at each step of the problem-solving process. However, the evaluation of the arithmetic capabilities of large language models remains an area that has received relatively little attention. In response, we introduce an extensive mathematics dataset called "MathQuest" sourced from the 11th and 12th standard Mathematics NCERT textbooks. This dataset encompasses mathematical challenges of varying complexity and covers a wide range of mathematical concepts. Utilizing this dataset, we conduct fine-tuning experiments with three prominent LLMs: LLaMA-2, WizardMath, and MAmmoTH. These fine-tuned models serve as benchmarks for evaluating their performance on our dataset. Our experiments reveal that among the three models, MAmmoTH-13B emerges as the most proficient, achieving the highest level of competence in solving the presented mathematical problems. Consequently, MAmmoTH-13B establishes itself as a robust and dependable benchmark for addressing NCERT mathematics problems.