Tiny Time Mixers (TTMs): Fast Pre-trained Models for Enhanced Zero/Few-Shot Forecasting of Multivariate Time Series

📄 arXiv: 2401.03955v8 📥 PDF

作者: Vijay Ekambaram, Arindam Jati, Pankaj Dayama, Sumanta Mukherjee, Nam H. Nguyen, Wesley M. Gifford, Chandra Reddy, Jayant Kalagnanam

分类: cs.LG, cs.AI

发布日期: 2024-01-08 (更新: 2024-11-07)

备注: Accepted at the 38th Conference on Neural Information Processing Systems (NeurIPS 2024)

🔗 代码/项目: HUGGINGFACE | HUGGINGFACE | HUGGINGFACE

💡 一句话要点

提出Tiny Time Mixers以解决多变量时间序列预测中的性能瓶颈问题

🎯 匹配领域: 支柱九：具身大模型 (Embodied Foundation Models)

关键词: 时间序列预测 零样本学习 少样本学习 迁移学习 轻量级模型 多变量数据 自适应分块 外生信号

📋 核心要点

  1. 现有的多变量时间序列预测模型在零/少样本学习中性能不足，且计算需求高，难以适应多样化的数据特征。
  2. 本文提出Tiny Time Mixers（TTM），通过轻量级架构和创新技术实现高效的迁移学习，专注于公共时间序列数据集的训练。
  3. TTM在零/少样本预测任务中表现优异，相较于现有基准提高了4-40%，并且能够在仅有CPU的机器上运行，降低了使用门槛。

📝 摘要（中文）

大型预训练模型在语言和视觉任务的零/少样本学习中表现出色，但在多变量时间序列预测中面临挑战。为此，本文提出了Tiny Time Mixers（TTM），一种紧凑的模型，专门针对公共时间序列数据集进行训练。TTM基于轻量级的TSMixer架构，采用自适应分块、多分辨率采样和分辨率前缀调优等创新，能够在不同数据集分辨率下进行有效的预训练。此外，TTM通过多层建模捕捉通道相关性，并在微调过程中注入外生信号。实验结果表明，TTM在零/少样本预测中比现有基准提高了4-40%，同时显著降低了计算需求，适合在资源受限的环境中使用。

🔬 方法详解

问题定义：本文旨在解决多变量时间序列预测中零/少样本学习的性能瓶颈，现有模型在计算效率和通道相关性捕捉方面存在不足。

核心思路：TTM通过轻量级的TSMixer架构和多种创新技术，优化了模型的预训练过程，使其能够在不同数据集分辨率下有效工作，同时增强了模型的迁移学习能力。

技术框架：TTM的整体架构包括自适应分块、分辨率采样和前缀调优等模块，能够处理多样化的时间序列数据，并在微调阶段引入外生信号以增强预测能力。

关键创新：TTM的主要创新在于其轻量化设计和多层建模策略，能够有效捕捉通道间的相关性，并在计算资源有限的情况下实现高效预测，这与传统的重型模型形成鲜明对比。

关键设计：TTM的参数设置从1M参数起步，采用了适应性分块和多分辨率采样策略，确保在不同数据集上进行有效的预训练，损失函数和网络结构经过精心设计以优化学习效果。

🖼️ 关键图片

fig_0
fig_1
fig_2

📊 实验亮点

TTM在零/少样本预测任务中相较于现有基准提高了4-40%的性能，同时显著降低了计算需求，能够在仅有CPU的机器上运行，极大地提升了模型的可用性和适用范围。

🎯 应用场景

TTM模型在多变量时间序列预测中的应用潜力巨大，适用于金融市场预测、气象数据分析和工业设备监控等领域。其轻量化特性使得在资源受限的环境中也能实现高效预测，推动了相关技术的普及和应用。

📄 摘要（原文）

Large pre-trained models excel in zero/few-shot learning for language and vision tasks but face challenges in multivariate time series (TS) forecasting due to diverse data characteristics. Consequently, recent research efforts have focused on developing pre-trained TS forecasting models. These models, whether built from scratch or adapted from large language models (LLMs), excel in zero/few-shot forecasting tasks. However, they are limited by slow performance, high computational demands, and neglect of cross-channel and exogenous correlations. To address this, we introduce Tiny Time Mixers (TTM), a compact model (starting from 1M parameters) with effective transfer learning capabilities, trained exclusively on public TS datasets. TTM, based on the light-weight TSMixer architecture, incorporates innovations like adaptive patching, diverse resolution sampling, and resolution prefix tuning to handle pre-training on varied dataset resolutions with minimal model capacity. Additionally, it employs multi-level modeling to capture channel correlations and infuse exogenous signals during fine-tuning. TTM outperforms existing popular benchmarks in zero/few-shot forecasting by (4-40%), while reducing computational requirements significantly. Moreover, TTMs are lightweight and can be executed even on CPU-only machines, enhancing usability and fostering wider adoption in resource-constrained environments. The model weights for reproducibility and research use are available at https://huggingface.co/ibm/ttm-research-r2/, while enterprise-use weights under the Apache license can be accessed as follows: the initial TTM-Q variant at https://huggingface.co/ibm-granite/granite-timeseries-ttm-r1, and the latest variants (TTM-B, TTM-E, TTM-A) weights are available at https://huggingface.co/ibm-granite/granite-timeseries-ttm-r2.