AutoGluon: Time Series Forecasting

AutoGluon can forecast the future values of multiple time series given the historical data and other related covariates. A single call to AutoGluon TimeSeriesPredictor’s fit() method trains multiple models to generate accurate probabilistic forecasts, and does not require you to manually deal with cumbersome issues like model selection and hyperparameter tuning.

Efficient Training Techniques for Transformers on a Single GPU

When training Transformer models on a single GPU, it’s important to optimize for both speed and memory efficiency to make the most of limited resources. Here are some key parameters and techniques to consider:

Mixed Precision Training

  • Use fp16 or bf16 mixed precision to reduce memory usage while maintaining most of the fp32 precision. Set fp16=True or bf16=True in TrainingArguments.
  • torch.bfloat16 or torch.float16 can reduce memory usage by 2-4x with minimal impact on model quality, allowing training with larger batch sizes.
  • bfloat16 is more numerically stable than float16 and recommended if supported by your GPU (e.g., Nvidia A100).
  • Mixed precision speeds up math-heavy operations like linear layers and convolutions.

Optimizer Choice

  • AdamW is the most common optimizer but has high memory usage.
  • Alternatives like adamw_hf, adamw_torch, adamw_apex_fused, adamw_anyprecision, or adafactor can be more memory efficient.
  • Fused optimizers like adamw_apex_fused from Apex or adamw_anyprecision fuse multiple operations for faster speed.
  • Fused Adam combines the Adam update’s elementwise operations into a single kernel, reducing memory access.

Gradient Accumulation

  • Accumulate gradients over multiple smaller batches before doing an optimizer step to emulate training with larger batch sizes. Set gradient_accumulation_steps in TrainingArguments.
  • Allows training with large effective batch sizes that don’t fit in GPU memory.
  • Increases training speed by reducing communication overhead but can slow down convergence.

Gradient Checkpointing

  • Trade off compute for memory by recomputing activations in the backward pass instead of storing them.
  • Speeds up training by allowing larger batch sizes but slows down each iteration.
  • Enable with gradient_checkpointing=True in TrainingArguments.

Efficient Data Loading

  • Use DataLoader(pin_memory=True) to speed up data transfer from CPU to GPU memory.
  • Set DataLoader(num_workers=N) to preload data with multiple worker processes.

Offload to CPU

  • Offload the optimizer state and model parameters to CPU when not in use to free up GPU memory. Set offload_optimizer=True and offload_param=True.
  • Speeds up training by allowing larger models and batch sizes.

TF32 on Ampere GPUs

  • Enable TF32 mode on Ampere or newer GPUs to automatically use the TF32 data type, which has the range of fp32 but precision of fp16. Set tf32=True in TrainingArguments.
  • Speeds up linear layers, convolutions, and matmuls with minimal accuracy impact.
  • Set torch.backends.cuda.matmul.allow_tf32 = True.

Flash Attention 2

  • Use Flash Attention 2 kernels integrated in the Transformers library to speed up attention computation and reduce memory usage.

The Trainer API in 🤗 Transformers supports most of these techniques via the TrainingArguments class. Experiment with combinations of these approaches to find the optimal tradeoff between speed, memory efficiency, and model quality for your specific model and hardware setup.



TinyTimeMixers (TTMs) are compact pre-trained models for Multivariate Time-Series Forecasting, open-sourced by IBM Research. With less than 1 Million parameters, TTM introduces the notion of the first-ever “tiny” pre-trained models for Time-Series Forecasting.

TTM outperforms several popular benchmarks demanding billions of parameters in zero-shot and few-shot forecasting. TTMs are lightweight forecasters, pre-trained on publicly available time series data with various augmentations. TTM provides state-of-the-art zero-shot forecasts and can easily be fine-tuned for multi-variate forecasts with just 5% of the training data to be competitive. 

Distilling Knowledge from Large LLMs: Fine-tuning Mistral with LoRA

As large language models (LLMs) continue to advance, there is a growing need to distill their knowledge into smaller, more efficient models suitable for real-world applications. One promising approach is knowledge distillation via fine-tuning using techniques like LoRA (Low-Rank Adaptation). In this article, we’ll dive into best practices for fine-tuning the 7B parameter Mistral model with LoRA.

The LoRA Advantage

Traditional fine-tuning updates all the weights of a pre-trained LLM, which can be computationally expensive and data-hungry, especially for large models. LoRA circumvents this by injecting trainable rank decomposition matrices into the LLM layers, enabling efficient adaptation to new tasks without modifying the original model weights.

Compared to full fine-tuning, LoRA requires significantly less compute and data, making it well-suited for fine-tuning models like Mistral. It has been shown to match or even exceed the performance of full fine-tuning on various tasks while using orders of magnitude fewer trainable parameters.

Selecting the Optimal LoRA Rank

The LoRA rank (r) determines the number of trainable parameters and directly impacts the model’s capacity to capture task-specific knowledge. A higher rank allows the model to better approximate the ideal fine-tuned weights, potentially improving performance. However, it also increases memory requirements and the risk of overfitting.

For Mistral, common ranks used are r=64 or r=128, though some have experimented with higher values like r=256 which can finetune around 8% of the model’s parameters. The optimal rank depends on the complexity of the task and dataset size – simple tasks may work well with lower r, while more complex ones may benefit from higher r.

Dataset Size and Quality

While LoRA is data-efficient compared to full fine-tuning, having sufficient high-quality training data is still crucial for achieving good performance. For a 7B model like Mistral, researchers recommend at least 50,000 examples for reasonable results, with 100,000+ examples often yielding better performance.

However, even smaller datasets of 1,000 – 10,000 carefully curated examples can be effective when using LoRA, outperforming full fine-tuning which requires much more data. Data quality and relevance to the target task are more important than sheer quantity – high-quality, curated datasets can outperform larger, noisier ones.

Using too little data (e.g. less than 1,000 examples) may lead to overfitting or poor performance. For very large datasets (>1M examples), full fine-tuning may be more effective than LoRA, depending on available compute resources.

Putting it All Together

So, what are the best practices for fine-tuning Mistral with LoRA? Based on current research, a good starting point could be:

  • LoRA rank (r) = 128
  • 10,000 – 100,000 high-quality, task-relevant examples

During training, it’s essential to monitor performance on a held-out validation set to select the best checkpoint and avoid overfitting. Additionally, increasing the LoRA alpha (lora_alpha) can help counteract a lower rank but may introduce instability.

Distillation Approaches

Beyond LoRA, researchers have explored various distillation approaches for transferring knowledge from large LLMs to smaller models:

  1. Reverse KL Divergence: Replacing the standard forward KL divergence loss with reverse KL can prevent the student model from overestimating low-probability regions of the teacher LLM’s distribution, making it more suitable for generative tasks.
  2. Multi-Task Learning with Rationales: Training the student on two tasks – label prediction and rationale generation, where rationales are intermediate reasoning steps extracted from the LLM teacher. This creates an explicit connection between inputs and outputs.
  3. Data Augmentation: Leveraging data augmentation to generate context-rich, skill-specific training data from the LLM teacher. This helps the student model approximate the teacher’s contextual abilities and ethical alignment.

The Future of LLM Distillation

As LLMs continue to grow in size and capability, techniques like LoRA and knowledge distillation will become increasingly important for making these models accessible and deployable across a wide range of applications.

By following best practices, leveraging the latest research, and adhering to legal and ethical considerations when working with LLM outputs, practitioners can effectively distill the knowledge from large models like Mistral into smaller, more efficient models tailored to their specific needs.

The possibilities for LLM distillation are vast, paving the way for a future where the power of large language models is available to everyone, regardless of computational resources.

Revolutionizing AI Efficiency: How Microsoft’s LLMLingua-2 is Changing the Game with 8x Less Memory

  • LLMLingua-2 is a novel compression technology developed by Microsoft Research, achieving state-of-the-art results with 8 times less GPU memory on tasks typically handled by models like GPT-4.
  • It introduces innovative approaches such as « Data Distillation, » « Bidirectional Token Classification, » and optimized compression objectives to efficiently compress prompts without losing key information.
  • The technology has shown superior performance across various language tasks and demonstrated remarkable generalization across different LLMs and languages, from GPT-3.5 to Mistral-7B and from English to Chinese.
  • Compared to existing prompt compression methods, LLMLingua-2 is 3 to 6 times faster, accelerates end-to-end inference by 1.6 to 2.9 times, and significantly reduces GPU memory usage by a factor of 8.
  • This advancement represents a significant step forward in making language AI more practical and scalable for real-world applications, demonstrating Microsoft Research’s leadership in the field.


Chronos: Learning the Language of Time Series

  • Chronos is a framework designed for pretrained probabilistic time series models.
  • It utilizes scaling and quantization to tokenize time series values into a fixed vocabulary.
  • Chronos trains transformer-based language model architectures (specifically, models from the T5 family with parameters ranging from 20M to 710M) using cross-entropy loss.
  • The models are pretrained on a mix of publicly available datasets and a synthetic dataset generated via Gaussian processes, enhancing generalization.
  • In a comprehensive benchmark involving 42 datasets, including both classical local models and deep learning approaches, Chronos models:
  • (a) significantly outperform other methods on datasets included in the training corpus;
  • (b) show comparable or occasionally superior zero-shot performance on new datasets compared to methods trained specifically on those datasets.
  • These results demonstrate the potential of pretrained models to leverage time series data across various domains for improving zero-shot accuracy on unseen forecasting tasks, suggesting a simplified approach to forecasting pipelines.

Unified Time Series Model

UniTS is a unified time series model that can process various tasks across multiple domains with shared parameters and does not have any task-specific modules.

Foundation models, especially LLMs, are profoundly transforming deep learning. Instead of training many task-specific models, we can adapt a single pretrained model to many tasks via few-shot prompting or fine-tuning. However, current foundation models apply to sequence data but not to time series, which present unique challenges due to the inherent diverse and multi-domain time series datasets, diverging task specifications across forecasting, classification and other types of tasks, and the apparent need for task-specialized models. 

We developed UniTS, a unified time series model that supports a universal task specification, accommodating classification, forecasting, imputation, and anomaly detection tasks. This is achieved through a novel unified network backbone, which incorporates sequence and variable attention along with a dynamic linear operator and is trained as a unified model. 

Across 38 multi-domain datasets, UniTS demonstrates superior performance compared to task-specific models and repurposed natural language-based LLMs. UniTS exhibits remarkable zero-shot, few-shot, and prompt learning capabilities when evaluated on new data domains and tasks. We will release the source code and datasets.

Unified Training of Universal Time Series Forecasting Transformers

  • Deep learning for time series forecasting traditionally uses a one-model-per-dataset approach, limiting potential advancements.
  • Universal forecasting introduces the idea of pre-training a single Large Time Series Model on a vast collection of datasets for diverse tasks.
  • Challenges in creating such a model include: cross-frequency learning, handling multivariate series with arbitrary variates, and varying distributional properties of large-scale data.
  • To overcome these challenges, novel enhancements to the time series Transformer architecture are introduced, creating the Masked EncOder-based UnIveRsAl TIme Series Forecasting Transformer (MOIRAI).
  • MOIRAI is trained on the Large-scale Open Time Series Archive (LOTSA), which contains over 27 billion observations across nine domains.
  • MOIRAI demonstrates competitive or superior performance as a zero-shot forecaster compared to full-shot models.

GPT in 60 Lines of NumPy

In this post, they implement a GPT from scratch in just 60 lines of numpy. We’ll then load the trained GPT-2 model weights released by OpenAI into our implementation and generate some text.


  • This post assumes familiarity with Python, NumPy, and some basic experience training neural networks.
  • This implementation is missing tons of features on purpose to keep it as simple as possible while remaining complete. The goal is to provide a simple yet complete technical introduction to the GPT as an educational tool.
  • The GPT architecture is just one small part of what makes LLMs what they are today.[1].
  • All the code for this blog post can be found at
  • Hacker news thread
  • Chinese translation

Text splitting

Large language models (LLMs) can be used for many tasks, but often have a limited context size that can be smaller than documents you might want to use. To use documents of larger length, you often have to split your text into chunks to fit within this context size.

This crate provides methods for splitting longer pieces of text into smaller chunks, aiming to maximize a desired chunk size, but still splitting at semantically sensible boundaries whenever possible.

Levels Of Text Splitting

Semantic text splitting library

Chunks Vizualizer