As we discussed three Low-Rank Adaptation (LoRA) fine-tuning methods in our latest AI 101 episode, we decided to add more useful articles to our library about different types of LoRA that can be used for various purposes. LoRA itself is one of the most popular and efficient fine-tuning techniques. It modifies only a small subset of the model's parameters, making fine-tuning easier and faster, and requiring less memory.
However, there are many ways to upgrade LoRA for specific scenarios, making it even more efficient for multitasking, working with diverse datasets, reducing memory usage, increasing stability, etc.
So, here are 11 types of LoRA that you can use to enhance fine-tuning:
Token 1.11: What is Low-Rank Adaptation (LoRA)? In this article we discuss in details how the LoRA method works and how it makes fine-tuning more efficient and less costly. → Read more
Topic 14: What are DoRA, QLoRA and QDoRA? Here we compare three fine-tuning methods that all address limitations of LoRA and improve memory efficiency in different ways. → Read more
Weight-Decomposed Low-Rank Adaptation (DoRA) makes fine-tuning easier and more stable. → Original paper
QLoRA is an efficient fine-tuning of quantized LLMs which drastically cuts down the memory needs. → Original paper
And finally, QDoRA combines DoRA and QLoRA approaches.
CURLoRA combines CUR matrix decomposition and LoRA to reduce trainable parameters and prevent forgetting in continual learning. It enhances performance and stability across datasets, especially with limited data. → Read more
DLP-LORA, a Dynamic Lightweight Plugin, uses a mini-MLP module to fuse LoRAs dynamically at the sentence level, reducing processing time. This improves accuracy and efficiency, making it ideal for multitask learning. → Read more
GraphLoRA, inspired by Low-Rank Adaptation (LoRA), which efficiently adapts GNNs to new graphs, enhances adaptability and preventing forgetting. It performs well across diverse datasets with minimal tuning. → Read more
SeLoRA, a Self-Expanding LoRA module, efficiently fine-tunes medical data for medical image synthesis. During training, it adds ranks to crucial layers to enhance synthesis quality. → Read more
Flat-LoRA addresses the issue of generalization in LoRA. It guides the model to solutions in a "flat" region of the full parameter space, using random weight perturbations for efficient fine-tuning. → Read more
FLoRA is a method for efficiently fine-tuning LLMs in federated learning (FL), where data stays local for privacy. It uses a stacking-based approach to combine local models without errors, allowing clients with different setups to work together smoothly. → Read more
UIR-LoRA is an image restoration framework that utilizes LoRA to handle various types of image degradation. It adapts a pre-trained generative model for specific tasks by transferring knowledge through LoRA and combines different LoRAs based on degradation similarity to effectively restore images with mixed degradations. → Read more
Bonus: Additional types of LoRA
X-LoRA dynamically combines pre-trained LoRA adapters to tackle diverse tasks by reusing neural network components. Applicable to any model without modification, it excels in scientific tasks such as protein mechanics and molecular design, providing adaptable, domain-specific knowledge and reasoning. → Read more
rsLoRA (rank-stabilized LoRA) suggests scaling adapters by the square root of the rank for faster learning, improving performance and allowing for better fine-tuning without increasing computational costs during inference. → Read more
VeRA, Vector-based Random Matrix Adaptation, reduces trainable parameters in LLMs by sharing low-rank matrices across all layers and learning small scaling vectors. It maintains performance while using fewer resources compared to original LoRA. → Read more
FFA-LoRA, Federated Freeze A LoRA, is specified for privacy-focused federated learning. It fixes certain matrices while tuning others to improve stability, offering better and more consistent performance compared to standard LoRA in federated learning tasks. → Read more
FedSA-LoRA is also created for federated learning. It uses two trainable matrices to update model weights locally. Matrice A learns general knowledge, while B captures client-specific knowledge. Based on this, FedSA-LoRA only shares A matrices with the server for updates. It shows better results than original LoRA and FFA-LoRA. → Read more