What are DoRA, QLoRA and QDoRA?

When specialized AI models, such as coders or those designed for specific domains, emerge, fine-tuning is crucial for boosting their performance and tailoring them to specific tasks. Low-Rank Adaptation (LoRA), which we previously discussed, is one such method, but it doesn't always deliver the best results in certain cases. To address these limitations, advanced methods like Weight-Decomposed Low-Rank Adaptation (DoRA), QLoRA, and QDoRA have been developed, offering enhanced performance. What are these methods, and how do they improve upon LoRA? Let’s explore!

In today’s episode, we will cover:

To tailor general models for specific tasks, we need to fine-tune them. This process usually involves retraining all the model’s parameters, but as models get bigger, this becomes more expensive and resource-intensive.

To solve this, Parameter-Efficient Fine-Tuning (PEFT) methods were developed, and one of them is LoRA (Low-Rank Adaptation). LoRA modifies fewer parameters for efficient fine-tuning while keeping the model architecture the same. However, it often doesn’t perform as well as full fine-tuning (FT) where all parameters are retrained. But why? 

NVIDIA and HKUST researchers, inspired by the concept of Weight Normalization, compared LoRA and FT. Weight Normalization separates the weight matrix into two components – magnitude (how large the change is) and direction (where the change is happening in the parameter space). This separation helps better understand how each method updates the model’s weights, allowing a more detailed comparison of the flexibility and precision in adjustments made by LoRA and FT.

They discovered that these two methods update the model in different ways": 

As LoRA lacks this flexibility, it may not always be able to make the same precise adjustments as FT. But full fine-tuning still requires retraining all parameters, which is computationally intensive.

To bridge the gap between efficiency and performance, NVIDIA and HKUST researchers developed a new method called DoRA (Weight-Decomposed Low-Rank Adaptation). DoRA separates the pre-trained weights into magnitude and direction, just like in their analysis, and fine-tunes both parts. It makes fine-tuning easier and more stable.

DoRA has been tested on various tasks, from NLP to tasks that involve both language and vision, and it consistently performs better than LoRA without adding extra inference time.

DoRA uses LoRA for the directional updates to keep things efficient but improves the model’s overall learning ability, making it behave more like full fine-tuning.

Here’s how it works step by step:

As DoRA method was made to solve the limitations of LoRA let’s look at its advantages:

To explore how good is DoRA’s performance researchers tested it on various benchmarks and different tasks and here’s what they have found out:

DoRA’s ability to fine-tune both the magnitude and direction of weight updates has significantly improved the flexibility of parameter-efficient fine-tuning methods, particularly when precise adaptations are required for specialized tasks. However, even though DoRA enhances performance without adding inference time, there remains another pressing challenge in fine-tuning large models: managing memory usage. This is where QLoRA comes in.

On the path to fine-tuning huge LLMs (like a 65 billion parameter one) and using less memory, researchers from University of Washington developed QLoRA: Efficient Fine-tuning of Quantized LLMs. It drastically cuts down the memory needs from 780 GB to less than 48 GB, allowing it to run on a single 48GB GPU, common in high-end consumer computers. Despite using less memory, it keeps the same high performance as regular fine-tuning methods and doesn’t slow things down. QLoRA is all about quantization.

Quantization reduces data by converting it to a simpler form, for example, changing 32-bit floats to 8-bit integers. This stores less detail but remains close to the original. Data is scaled to fit properly in the new range.

QLoRA implements a few memory-saving techniques which are steps of its working process:

“On average, for a blocksize of 64, this quantization reduces the memory footprint per parameter from 32/64 = 0.5 bits, to 8/64 + 32/(64 · 256) = 0.127 bits, a reduction of 0.373 bits per parameter.” – From Original QLoRA paper.

The Guanaco model family, trained using QLoRA, demonstrated great performance:

Even with QLoRA, loading the model onto GPUs still requires a lot of memory. Can something be better at reducing memory use than QLoRA? The answer is QDoRA.

QDoRA replaces LoRA with DoRA in the QLoRA framework, introducing a more advanced approach to fine-tuning by decomposing the weight matrix into two components: magnitude and direction. This approach was introduced in a recent project ”Efficient finetuning of Llama 3 with FSDP QDoRA” led by Kerem Turgutlu. It also uses FSDP (Fully Sharded Data Parallel) to split the model and train it across multiple GPUs, making it even more efficient.

This allows for more precise weight adjustments during fine-tuning, unlike QLoRA, which applies uniform changes. By combining this decomposition with quantization, QDoRA enhances the flexibility of full fine-tuning while retaining the memory efficiency of QLoRA.

This approach enables QDoRA to outperform QLoRA in various tasks by making more granular updates to model weights, thus achieving closer performance to full fine-tuning without significantly increasing memory or computational requirements.

In tests, QDoRA fine-tuned LLaMA2-7B and LLaMA3-8B on Orca-Math dataset, outperforming both QLoRA and full fine-tuning with less memory use. This makes QDoRA ideal for developers with limited GPU resources.

Llama3, when enhanced with either QDoRA or full fine-tuning, significantly exceeds the performance of both QLoRA and Llama2 during math data training.

In summary, QDoRA brings together the best of both worlds – efficient memory usage and more adaptable, performance-oriented fine-tuning.

We discussed several methods that work better than LoRA method for fine-tuning: 

Every task needs its specific approach, and when in one case LoRA will be enough for efficient fine-tuning, in the other case (with larger models and with huge memory) only QDoRA can work efficiently.

Resources