We want LLMs to be as fast and accurate as they can, so chasing high inference speed is an important and non-trivial job for developers. Since LLMs run on hardware, there are two things you can rely on for processing speed:

The thing is that LLMs are more memory-bound than compute-bound. We are in the era when generating a token requires only small matrix-vector multiplications, but involves loading large amounts of data from memory, and this process is constantly repeated. For LLM inference, memory is a bigger issue than math (hardware compute), which gives us a lot of research trying to overcome these memory limits.

One of them is a recent study from UC Berkeley, FuriosaAI, ICSI, and LBNL on XQuant – a method that can cut memory use up to 12 times by adding a little bit more compute cost. XQuant and its variation XQuant-CL bypass typical techniques like KV cache quantization that usually lead to accuracy drop.

So today, starting from KV caching, we will follow the journey of the XQuant creators to see how this new technique actually works and what it is really capable of. Join us!

In today’s episode, we will cover:

Today we focus on how to deal with large memory use. For short texts, the model weights are the main memory load, while for longer texts, the bigger problem is the Key-Value (KV) cache. It stores the representation of the whole sequence for self-attention to help the model track the full context, growing linearly with input length, which causes limitations.

One of the most popular method is to compress the KV cache through quantization. But first of all, let’s look at what KV cache exactly is.

In transformer models, every time a token is processed, the attention mechanism needs Keys (K) and Values (V) to figure out how much each previous token should influence the current one. Each token is represented by these three vectors:

The model’s attention mechanism compares a token’s Query with all previous Keys. The resulting similarity scores are used to weight the corresponding Values and to produce a context-aware representation of the token.

During autoregressive text generation, the model typically recomputes all K and V matrices from scratch for every new token. However, this is redundant and computationally expensive, since the cost grows with the length of the sequence.

Overall, the KV cache is a memory structure that stores the previously computed Keys and Values, so they don’t need to be recalculated at each step of decoding.

And what is KV caching?

KV caching is the process of using the KV cache during inference. Instead of recalculating attention against the entire sequence of tokens every time, the model only does new work for the new tokens:

In this case, the cache grows with each step and attention is computed much faster.

As we have already mentioned KV cache grows linearly with input length, exploding memory usage. There are two common ways to compress KV cache.

The first method is quantization. Compressing the KV cache using quantization means representing it with fewer bits. Thanks to this, more tokens can be cached, enabling longer context windows and reducing memory use. However, when the quantization goes too low, like 2–3 bits, model accuracy drops sharply.

Another approach is to shrink the KV cache using low-rank decomposition – compressing the KV cache into smaller spaces. This means breaking large matrices into smaller, compressed forms. However, this method also comes at an accuracy cost: it risks throwing away important information and is mathematically heavy because of compression and decompression that happens across different ranks.

Another interesting idea came from OpenMachine and is called Slim Attention. It stores only Keys and uses math tricks to recover Values from them. But this requires unstable matrix inverses and also doesn’t work well with Rotary Position Embeddings (RoPE) or Grouped-Query Attention (GQA).

So all these methods point out a big issue:

That’s why we need something that can save memory better, be more accurate and easier to compress. And here is one of the freshest solutions. Another direction attacking the same memory wall is conditional memory in LLMs — a selective activation principle where models choose what knowledge to access rather than loading everything at once.

XQuant is a new method proposed by a group of researchers from UC Berkeley, FuriosaAI, ICSI, and LBNL that trades some extra computation for much less memory usage when running LLMs and breaks, as the researchers call it, the Memory Wall. They follow the idea: GPUs can theoretically do huge amounts of math, but they can’t feed themselves data fast enough; so the logical solution is to reduce memory operations, even if it costs additional compute.

Instead of storing the usual KV cache, XQuant quantizes and stores only the layer input activations, called X. Then, when needed, it rematerializes (recalculates) Keys and Values from X on the fly during inference. This method allow not to store everything in memory, but throw away some data and recompute it later when needed. Let’s look at the inner side of this process. It’s fascinating →

XQuant stores less and recalculates more, and here is how it does it in several steps:

This is a good tradeoff because storing X requires only about half the memory compared to standard KV caching. Researchers didn’t stop on this version of their method and also created something more interesting.

Developers of XQuant noticed that X looks similar across layers. This happens because each layer only slightly refines the input before passing it along – it is called residual stream.

This hint encouraged researchers to make their approach even smarter, creating XQuant-CL that doesn’t store each full X, but stores only the differences (deltas) between them. This makes extreme compression possible:

More precisely XQuant-CL’s step-by-step workflow looks like this:

Even though XQuant-CL uses simple uniform quantization, it performs better than more complicated KV compression methods.

In pursuit of the best efficiency across different model types, researchers pushed XQuant and XQuant-CL’s capabilities further, and their next stop was the support of Grouped-Query Attention (GQA). Why is it so necessary?

The standard attention type in LLMs is Multi-Head Attention (MHA) where every query head has its own Keys and Values. Many efficient models, such as Llama, Mistral, Gemma, Qwen, use Grouped-Query Attention (GQA) with several heads that share the same Keys and Values, which already reduces the memory size of the KV cache.

GQA is a great benefit for AI models, but plain XQuant has some issues with it. The input activations X are large, while Keys and Values in GQA are much smaller. Simply caching X, or even deltas in XQuant-CL, instead of KV could actually use more memory than standard KV caching. This issue just cancels the benefit of XQuant for GQA-based models, and it is not what we need.

To solve this problem, XQuant and XQuant-CL use Singular Value Decomposition (SVD) offline on the Key and Value projection matrices (Wk and Wv). Here is how it works:

An interesting detail: in the down-projected Key space, all outliers cluster in the first channel, which works good for effectiveness, while Values are best handled per-token.

With this method, the memory footprint now matches GQA’s KV cache size, but the down-projected X is easier to quantize than KV directly, so accuracy stays higher at the same bit width.

XQuant-CL for GQA-based models have the same benefits as XQuant version. Overall, since the heavy SVD step is done offline, there’s no extra runtime cost.

This is all about the inner side of the XQuant and XQuant-CL workflow, but what do we end up with?

From the very beginning of employing XQuant, we witness immediate gains – just by switching to storing X (one tensor per layer), memory use is cut in half compared to the normal KV cache.

Then better gains come with quantization. By compressing X into fewer bits, the memory savings grow up to ~7.7×, while model quality barely drops: <0.1 change in perplexity compared to FP16. The model stays almost as accurate as the original.

Going further, XQuant-CL version takes advantage of the fact that X values look similar across layers. It compresses even more, reaching up to 12.5× memory savings with very little accuracy loss.

As for the GQA models (Llama-3.1-8B, Mistral-7B), results for XQuant are the following:

So, summing up the advantages of all XQuant methods, we get →

But after all that, a few problems dare to spoil the picture (a little bit).

Still, in memory-constrained scenarios where accuracy matters, XQuant offers an excellent balance: near-baseline accuracy with massive memory savings.

The big work by researchers from UC Berkeley, FuriosaAI, ICSI, and LBNL that we explored today gives us a new angle on dealing with memory limitations, avoiding the classical direct KV cache compression:

We may say that familiar, time-tested approaches like KV caching are outdated, but they are still a solid starting point and a baseline for the current and future breakthrough. XQuant and XQuant-CL take a cleaner route to save more memory with fewer downsides. Which method to use depends on your specific needs – but XQuant is there when you are chasing high accuracy and your number one mission is saving memory at large scale.

Sources and further reading

Resources from Turing Post