The Evolution of LoRA: 15+ Variants You Should Know

LoRA (original)
The foundations of the method that makes fine-tuning more efficient and less costly → Explore more

QLoRA

An efficient fine-tuning of quantized LLMs which cuts down memory needs up to 20x times → Explore more

DoRA

Weight-Decomposed Low-Rank Adaptation (DoRA) makes fine-tuning easier and more stable by separating magnitude and direction of weight updates. → Explore more

QDoRA
Combines QLoRA and DoRA for the best trade-off: memory + stability → Explore more

rsLoRA (Rank-Stabilized)

Suggests scaling adapters by the square root of the rank for faster learning. Improves performance and allows for better fine-tuning without increasing computational costs during inference. → Explore more

VeRA (Vector-based Random Adaptation)
Shares matrices across layers and learns scaling vectors, using much fewer resources compared to standard LoRA. → Explore more

SingLoRA (Single-Matrix LoRA)
Simplifies LoRA by using only one small matrix instead of usual two, and multiplying it by its own transpose (like A × Aᵀ). It uses half the parameters of LoRA and avoids scale mismatch between different matrices → Explore more

Sensitivity-LoRA

Dynamically assigns ranks to weight matrices based on their sensitivity, measured using second-order derivatives → Explore more

ARD-LoRA (Adaptive Rank Dynamic)
Adjusts the rank of LoRA adapters dynamically across transformer layers and heads by learning per-head scaling factors through a meta-objective. It balances performance, efficiency, using fewer parameters and reducing memory use.→ Explore more

Mixture-of-LoRA-Experts

Adds multiple low-rank adapters (LoRA) into a model’s layers, and a routing mechanism activates the most suitable ones for each input. This lets the model adapt better to new unseen conditions. → Explore more

X-LoRA

Dynamically combines pre-trained LoRA adapters to tackle diverse tasks by reusing neural network components. Applicable to any model without modification, it excels in scientific tasks such as protein mechanics and molecular design, providing adaptable, domain-specific knowledge and reasoning. → Explore more

AutoLoRA
Uses multiple LoRA adapters for customizing large image models. It retrieves relevant LoRAs based on semantic similarity to a text prompt and dynamically combines them using a gated fusion mechanism across layers and timesteps. → Explore more

LAG (LoRA-Augmented Generation)

Dynamically selects and applies relevant adapters per token and layer without extra training or data. LAG improves performance on knowledge-intensive tasks and can also integrate with retrieval-based methods like RAG when external data is available. → Explore more

T-LoRA (Timestep-Dependent)

A timestep-dependent LoRA method for adapting diffusion models with a single image. It dynamically adjusts updates and uses orthogonal initialization to reduce overlap, achieving better fidelity–alignment balance than standard LoRA → Explore more

Text-to-LoRA

Generates LoRA adapters directly from natural-language task descriptions. A hypernetwork reads the task text and produces low-rank weight updates for a frozen model. This removes the need for task-specific training, allowing instant adaptation to new or unseen tasks, while also compressing many task behaviors into a single generator model. → Explore more

Doc-to-LoRA

This one is similar to Text-to-LoRA but converts a document into a LoRA adapter in a single forward pass using a hypernetwork. → Explore more

LoRA-Squeeze

It is a way to shrink LoRA adapters after training for easier deployment. You don't need to rank upfront, you can train with a larger rank first, then compress it down using methods like randomized SVD. → Explore more

Mixture of Adapters (MoA)

Builds a mixture of heterogeneous Parameter-Efficient Fine-Tuning (PEFT) adapters, instead of relying on many identical LoRA experts. The point is not just to add more experts, but to add different kinds of experts with complementary capacities. → Explore more

Also, check out this article for more about the full advanced LLM fine-tuning stack with LoRAs.