Token 1.11: What Is Low-Rank Adaptation (LoRA)? A Guide to LLM Fine-Tuning

TL;DR: LoRA fine-tunes large models by freezing the original weights and training small low-rank update matrices. It cuts trainable parameters, optimizer memory, storage, and task-switching cost while preserving strong quality. Use it for behavior or style adaptation; use RAG for changing facts, and full fine-tuning when low-rank capacity is insufficient today.

As the utilization of Large Language Models (LLMs) intensifies across various domains, the concept of fine-tuning garnered significant attention. Particularly in the context of billion-parameter models, fine-tuning is often seen as a resource-intensive endeavor. This has led to a focus on optimization methodologies, one of which is Low-Rank Adaptation (LoRA).

Definition: Low-Rank Adaptation (LoRA) is a parameter-efficient fine-tuning method that keeps a pretrained model’s original weight matrices frozen. Instead of updating every parameter, it learns two much smaller matrices whose product approximates the needed weight change. These adapters can be trained, stored, combined, swapped, or merged cheaply, making model customization practical on limited hardware without adding inference latency after merging today.

And today we are going to discuss it in detail. However, to fully appreciate its significance, it's crucial first to understand the necessity of fine-tuning in the LLM landscape and where it stands in comparison to other adaptation techniques.

Let’s dive in:

Adapting LLMs to your specific needs is key to making the most out of these powerful tools in your business. Here are some straightforward ways to do this:

We've explored chain-of-thought as part of prompt techniques and RAG, finding that they are more straightforward to implement compared to fine-tuning. Prompt techniques (especially prompt engineering) are relatively simple, requiring only a few examples to guide the model. RAG, while also less demanding, offers the benefit of integrating domain-specific data without the need to retrain the model. However, fine-tuning, despite its higher cost in terms of computational power, memory requirements, time, and expertise, is sometimes the only viable option for certain tasks. This is particularly true in scenarios where the level of customization and accuracy needed goes beyond what prompt engineering and RAG can provide.

Given these scenarios, fine-tuning stands out for its ability to closely tailor a model to specific, often intricate requirements that broader methods can't adequately address.

So, the question arises: is there a way to make fine-tuning more efficient and less costly? One of the ways to do it is Low-Rank Adaptation (LoRA). Ready? Let’s go.

In 2018, researchers asked themselves a question: how many parameters (or model weights) are actually necessary for a model to perform well on various tasks? Their approach was innovative – they took random subsets of the model's original parameters and gradually increased their size. They were looking for the smallest subset size at which the model still worked effectively, a point they termed the 'measured intrinsic dimension' of a problem.

What they discovered was quite astonishing. In some instances, the intrinsic dimension was significantly lower than the total number of parameters in the original model. This insight opened the door to network compression, suggesting that models could be made smaller, yet remain effective. (Check ‘Large vs Small in AI: The Language Model Size Dilemma’.)

Building on this idea, in 2020, Aghajanyan et al. took a step further. They demonstrated that many pre-trained models actually have a very low intrinsic dimension. This meant that there exists a more compact way to reparameterize* these models – a 'low dimension reparameterization' – that's just as effective for fine-tuning as using the full set of parameters.

This work paved the way for the development of Low-Rank Adaptation (LoRA). The creators of LoRA were inspired by the concept that the changes in a model's weights during adaptation also exhibit a low intrinsic dimension.

LoRA essentially proposes a method to adapt and fine-tune large models in a more resource-efficient way. The core idea is to modify only a select subset of the model's weights, rather than the entire network, making the fine-tuning process more feasible and less demanding in terms of resources.

Here's a more streamlined explanation of how LoRA works:

In one of our previous practical articles How to Leverage Open-Source LLMs in Your Project, we talked with practitioners about leveraging LLMs in your project. Using LoRA or other type of parameter-efficient fine-tuning was one of the general recommendations in case fine-tuning is necessary.

Let’s briefly list the benefits of LoRA:

Useful links

The LoRA family has grown well past these three. For the full picture — DoRA, rsLoRA, LoRA+, VeRA and others — see our overview of 15+ LoRA variants.

QLoRA keeps a quantized base model in memory while training LoRA adapters, sharply reducing GPU memory use. It is a strong default when the base model does not fit comfortably at higher precision, although quantization choices can affect training stability and final quality.

AdaLoRA allocates the rank budget dynamically, giving more adapter capacity to important weight updates and pruning less useful directions. It is helpful when a fixed rank wastes parameters or when training must meet a strict parameter budget.

S-LoRA is a serving system for running many LoRA adapters over shared base models. It focuses on batching, memory management, and adapter scheduling, making it relevant for multi-tenant deployments where many customized models must share the same accelerators.

No. Diffusion-model users train LoRA adapters for visual styles, characters, products, and concepts without storing a full checkpoint. Vision transformers and multimodal models use LoRA for classification, segmentation, detection, and cross-modal adaptation. The same idea applies whenever a large matrix can express a useful task update through smaller low-rank factors.

LoRA is not automatically the best choice when the new task requires a large change in model capacity, the training data is poor, target layers are chosen badly, or the adapter rank is too small. It can also underperform full fine-tuning on major distribution shifts, and stacking many adapters can complicate serving. Evaluate against a prompt-only, RAG, and full-tuning baseline before standardizing on it.

To sum up, LoRA emerges as a crucial development in the fine-tuning of LLMs, particularly for models with extensive parameters. It addresses key challenges associated with traditional fine-tuning methods, notably in terms of computational and memory requirements.

However, it's important to understand that fine-tuning, made more efficient by LoRA, is just one of several adaptation techniques available for LLMs. Other methods, such as prompt engineering and RAG, provide alternative approaches. Each technique offers different levels of customization and efficiency, and their effectiveness can vary depending on the specific use case.

Thank you for reading, please feel free to share with your friends and colleagues. In the upcoming weeks, we are announcing our referral program 🤍

LoRA is used to adapt a pretrained model to a new task, domain, response style, visual concept, or user preference without retraining all of its weights. Common uses include instruction tuning, domain assistants, code models, image-generation styles, vision models, and inexpensive per-customer adapters.

RAG retrieves external information at inference time, so it is best for current, private, or citable knowledge. LoRA changes a model’s learned behavior through training, so it is better for style, format, task patterns, or domain reasoning. Many production systems use both.

In natural language processing, LoRA is a parameter-efficient fine-tuning method that freezes the base language model and trains small low-rank matrices inside selected layers. The resulting adapter captures task-specific changes while remaining much smaller than a complete copy of the model.

Choose target layers and rank, freeze the base model, add two trainable low-rank matrices to each target weight, train only those matrices on task data, evaluate the adapter, and then either load it alongside the base model or merge its update into the model for deployment.

No. LoRA can adapt any architecture with suitable weight matrices. It is widely used with diffusion models for image styles and subjects, vision transformers for classification and segmentation, multimodal models, speech systems, and other large neural networks.

Previously in the FM/LLM series:

Turing Post