LORA: Low-Rank Adaptation of Large Language Models implemented using PyTorch

Overview

LORA is a technique for adapting large language models using low-rank approximation, implemented using PyTorch. This repository contains the code for applying LORA to the MNIST dataset, as well as explanations and examples to help you understand and use the technique effectively.

Background

LORA (Low-Rank Adaptation) is a method for reducing the computational complexity of large language models by decomposing weight matrices into low-rank approximations. This allows for more efficient training and inference, especially in scenarios where computational resources are limited. MNIST Dataset

The MNIST dataset consists of 28x28 pixel grayscale images of handwritten digits (0-9). It is commonly used as a benchmark dataset for image classification tasks and provides a suitable environment for testing the effectiveness of LORA.

Implementation

The implementation of LORA in this repository is based on PyTorch, a popular deep learning framework. Here's a detailed overview of the key components:

1. Matrix Decomposition:

• The weight matrices of the neural network model are decomposed into low-rank matrices using techniques like Singular Value Decomposition (SVD) or Alternating Least Squares (ALS).
• The decomposition process helps in reducing the computational complexity of the model while preserving important features.

2. Incremental Update:

• LORA allows for incremental updates to the low-rank matrices as new data is encountered during training.
• This is achieved by adding or modifying certain components of the decomposition to adapt to the new information without recalculating everything from scratch.
• Incremental updates help the model stay up-to-date with changing data distributions and improve its performance over time.

3. PyTorch Integration:

• The implementation leverages PyTorch's computational graph and autograd capabilities to efficiently compute gradients and perform backpropagation during training
• Custom layers and modules are implemented using PyTorch's nn.Module interface, making it easy to integrate LORA into existing neural network architectures.

4. Training Loop:

• The training loop orchestrates the process of loading data, forward and backward passes through the network, parameter updates, and evaluation.
• Techniques such as mini-batch processing, learning rate scheduling, and early stopping may be employed to improve training efficiency and performance.

Files:

• lora.ipynb = contains the main file with the implementation of LoRa.
• svd.ipynb(Singular value decomposition) = contains the math behind the LoRa implementation.

Experiment and Customize:

Feel free to experiment with different hyperparameters, architectures, and datasets to see how LORA performs in various scenarios.Additionally, you can customize the implementation to suit your specific requirements by modifying the code or extending it with additional features.

Acknowledgments

I would like to thank the open-source community for their contributions to PyTorch and other libraries that made this project possible. Additionally, I acknowledge the researchers and developers who have contributed to the advancement of techniques like low-rank approximation and its applications in machine learning. Thanks to Umar Jamil for the tutorial on implementation of LoRA. https://github.com/hkproj

Images

Fine Tuning

Problems with Fine Tuning

LoRA

LoRa paper implementation

Total number of parameters

Parameters introduced by LoRa addition

Error rate for the identification of each digit without the addition of LORA

Error rate for the identification of each digit with the addition of LORA

Total number of parameters to be saved drastically reduced