LLM Inference Latency and Throughput: How to Optimize

_{Was this email forwarded to you? Forward it also to a friend or a colleague! Sign up}

New to inference? Start with the basics: what LLM inference is and how it works.

Powerful reasoning models that take significant time to generate a final response remain a major focus in the AI community. They use test-time scaling to achieve state-of-the-art performance, often at the cost of slower response speeds. Our article on test-time compute gained a lot of interest from our readers, which is why we decided to dive deeper into a closely related topic – inference.

All chatbots, image generators, voice assistants, and recommendation systems rely on inference; it’s what users actually interact with. That’s why understanding the core aspects of inference is important for everyone, not just developers. Once a model is trained, inference is the stage where it demonstrates the knowledge it has learned. Making inference faster and enabling models to process many inputs simultaneously are among the most important challenges in optimization. Today, we'll look at what actually drives inference latency and throughput – the metrics that matter, where time is spent inside the model, which levers move them, and what current hardware and serving stacks deliver

This article isn’t meant to be exhaustive but aims to clarify why inference matters so much in AI today.

What’s in today’s episode?

During training, a model learns essential patterns from data. Once it's properly trained, it's time for the model to demonstrate its acquired “knowledge” in real-world tasks. The process that begins with prompting and ends when the model generates an answer to a query is called inference. In other words, inference is the process of using a trained model to make predictions or generate outputs.

IBM Research has noted that up to 90% of an AI model’s life may be spent in inference mode. That refers to the model’s deployed lifetime, not a measured share of lifetime compute. The larger point still holds: once a model serves many users, inference can dominate its operating cost and carbon footprint.

We even started to talk about new scaling laws, thanks to inference. In one of his presentations, NVIDIA CEO Jensen Huang has distinguished three scaling laws shaping modern AI:

Jensen Huang emphasized that inference scaling is becoming increasingly critical. Models now perform complex, multi-step reasoning at inference – an approach called "test-time compute" or "inference scaling." By significantly increasing the computational resources allocated at this stage, AI systems achieve better performance, accuracy, and reliability, particularly in challenging applications requiring careful thought or nuanced reasoning.

To be fluent in AI inference, we need to understand its main aspects, such as the concepts used to measure the effectiveness of inference from different perspectives and how the inference process works within the model. Let's start with the concepts first.

Key concepts of inference include:

However, there is often a trade-off: optimizing for latency can hurt the throughput and vice versa. Therefore, the important task for developers is to balance them effectively, achieving minimized latency and maximized throughput. Both sides of that trade-off have to be measured on real traffic, not estimated: see how to monitor LLMs for the KPIs to track, and the open-source tools that capture TTFT and TPOT per request.

Other important concepts of inference, which developers also pay attention to, are:

This is what we can observe outside the model to estimate the inference process. However, we are here to explore inference in a bit more technical depth, so →

Here’s a simplified step-by-step flow of full inference process in LLMs:

These steps are hidden from the user, but each one affects whether inference is fast, reliable, and economical. The prefill/decode split is especially important because it connects the user-facing metrics to what the serving system is actually doing.

In diffusion models, the inference process is different. The initial input is random noise. Prompts are used to guide the generation toward a desired output, such as a specific image. The generation process involves iterative denoising steps, where the model gradually transforms the noise into coherent data. The number of denoising steps can vary, for example from 25 to 1000, depending on the sampling strategy and quality-speed tradeoff. Techniques like DDPM, DDIM, or score-based sampling define how the noise is removed over time. Because of the many forward passes required, inference in diffusion models tends to be slower compared to transformers, which generate outputs more directly.

Here we’ll look at several types of inference that are mostly used. In practice, they are often combined together depending on the purpose of the system.

Systems with real-time inference make predictions on-demand, in response to live data or user actions. They deliver low-latency results, essential for time-sensitive tasks, such as fraud detection or medical alerts. Real-time inference is also used in voice assistant or self-driving car. It provides immediate feedback and dynamic updates, but requires always-on infrastructure, careful scaling to handle simultaneous users, and high system reliability.

The opposite of real-time inference is batch inference, where large sets of inputs are collected and processed in bulk, usually on a schedule. It is an offline inference type used for tasks like processing millions of records. It maximizes throughput and efficiency, but can’t process queries immediately.

Cloud inference uses centralized servers, sending data from user devices to the cloud for processing. It allows powerful, scalable computation using high-end GPUs and TPUs, ideal for large or complex models. It provides easy updates and efficient resource sharing. However, there are some drawbacks, such as network latency, high cost and the rise of privacy concerns as data leaves the device. It is often paired with edge inference for optimal performance.

Edge inference runs AI models directly on local devices like phones, IoT sensors, or drones, enabling real-time processing at the source without sending data to the cloud. It is essential for fast and private AI decisions. However, edge devices have limited compute power, and it’s challenging to run large models without optimization on them. Updating models on multiple devices is also complex, and scaling needs more hardware deployment.

All these types can be mixed up together, for example, the system can do quick initial inference on the edge and send data to the cloud for more in-depth analysis, or perform real-time inference in the cloud by deploying servers in many regions.

As you can see from the example of these inference types, inference encounters many limitations. Various optimization techniques are widely used to address them.

Here are the main techniques for making inference faster and more efficient. Continuous batching is a separate serving-side optimization: production servers admit new requests into an in-flight batch as capacity becomes available, which raises hardware utilization and throughput but can increase per-request latency. It combines with the techniques below rather than replacing them.This raises hardware utilization and throughput, but larger or more aggressive batches can increase per-request latency.

These techniques are widely used in current AI systems and have even become traditional methods for achieving faster inference. But what are the actual trends in AI inference?

2026 update: inference work has shifted from isolated speed records toward end-to-end serving systems. The practical gains now come from combining continuous batching, KV-cache reuse, quantization, speculative decoding, and, at larger scale, separating prefill from decode. Because performance depends on the model, hardware, precision, batch size, context length, and latency target, the vendor figures below are useful snapshots—not universal rankings.

Reasoning models have made inference optimization more urgent, not less. Longer chains of thought increase output length and test-time compute, so serving systems now have to balance interactive latency with aggregate throughput and cost. The field’s center of gravity has therefore moved from “which chip is fastest?” to hardware-software co-design: precision formats, kernels, schedulers, cache management, networking, and disaggregated serving all matter.

Cerebras makes the same point from the wafer-scale side. In August 2026, it introduced CS-4, built around WSE-3T, and reported more than 4,400 tokens per second per user on GPT-OSS-120B. That is a vendor benchmark for a specific model and configuration—not a universal comparison—but it shows how wafer-scale hardware can reduce decode latency. The gain comes from hardware, compiler, and model-serving improvements together, not from software alone.

NVIDIA’s March 2025 DeepSeek-R1 result needs similar context. The widely quoted 36× throughput and 32× cost-per-token gains compare a sequence of H100 FP8, H200, and B200 FP4 configurations at maximum concurrency; they are combined hardware-and-software gains, not a clean Blackwell-versus-Hopper comparison. In the same post, the more comparable DGX B200 FP4 result delivered over 3× the inference throughput of DGX H200. NVIDIA also reported over 250 tokens per second per user on one eight-GPU DGX B200 at concurrency 1—a low-latency configuration that gives one request the whole system.

NVIDIA Dynamo is the serving layer in this story. It separates prefill and decode onto different GPU pools, routes requests toward reusable KV cache, and reallocates workers as traffic changes. NVIDIA’s original “up to 30×” claim applied specifically to projected DeepSeek-R1 serving on GB200 NVL72 with a 32K-token input and 8K-token output—not to every workload. By Dynamo 1.0 in 2026, NVIDIA reported up to 7× higher request throughput for DeepSeek-R1-0528 on GB200 NVL72 when disaggregated serving was combined with wide expert parallelism.

These examples are snapshots, not rankings. Across systems, the durable trend is co-design: hardware speed matters, but the serving stack decides whether that capability becomes lower latency, higher throughput, or lower cost.

In addition to hardware leaps, some of the biggest breakthroughs are happening in the software layer – changing not just how fast inference can be, but how smart it can become.

Take speculative decoding. A smaller draft model proposes several tokens, and the target model verifies them in parallel. When the draft predicts well, the system accepts multiple tokens per target-model step without changing the output distribution. The speedup depends on acceptance rate, model pair, sequence length, and hardware; it is a technique, not a guarantee that latency will be cut in half.

Another important open-source development is vLLM. It began at Berkeley in 2023 and is no longer new; it is now a mature serving engine built around PagedAttention, continuous batching, prefix caching, quantization, and distributed execution. Its original paper reported up to 24× higher throughput than an unoptimized Hugging Face Transformers baseline, but that historical result should not be read as a current comparison with modern serving engines.

Faster inference can also be spent on more work rather than a shorter wait. A system may sample several candidate solutions, verify intermediate steps, or allocate more tokens to reasoning. That can improve answer quality, but only when the extra search and evaluation are well designed; raw token speed alone does not make a model reason better.

So while inference optimization has always been about speed, the new era is also about strategic deliberation. Faster systems do both: respond quicker and think better.

Today, we explored a crucial topic that serves as the "face" of all AI systems: AI inference. This is what everyone experiences when interacting with AI. This aspect of model functioning drives progress in hardware and various techniques that can enhance inference effectiveness. We now face a two-sided challenge: reasoning models like DeepSeek R1 are increasing inference time, while researchers continue to find more advanced ways to speed up inference despite these increased demands. And it's like endless progress because we all still need systems that are as fast as possible.

There is much much more to cover in inference. Please suggest your topics in the comment section.

Sources and further reading

Resources for further reading:

Turing Post resources

LLM latency is the delay between sending a request and receiving the model’s output. In streaming applications, it is commonly evaluated using Time to First Token (TTFT) and Time Per Output Token (TPOT). Model size, context length, output length, hardware, batching, and serving infrastructure can all affect latency.

LLM latency can be reduced with KV caching, quantization, speculative decoding, model distillation, optimized hardware, and efficient request scheduling. Reducing unnecessary context and choosing smaller models can also improve response times. In production, developers typically balance latency against throughput, cost, and model quality rather than optimizing speed alone.