Token 1.19: How to optimize LLM Inference

Updated September 1, 2026. This article was first published in February 2024. We kept its original structure and explanations, while updating the parts that changed most: serving engines, ONNX Runtime GenAI, quantization, and hardware choices.

LLM inference optimization is a balancing act between latency, throughput, memory use, and cost. There is no single setting that improves all four. A larger batch may raise throughput while slowing an individual request; lower precision may reduce memory use while affecting quality; more parallelism may help one model and waste resources on another.

So how do you optimize LLM inference in practice? Begin with a realistic baseline. Measure time to first token, inter-token latency, throughput, memory use, and cost per request. Then choose the engine and hardware, apply quantization where quality allows, reuse the KV cache, continuously batch compatible requests, and test speculative decoding or parallelism against that baseline.

If you are someone who has been working in the field of machine learning (ML), then you must be aware of the fact that ML models, be they conventional ML models or Foundation Models (FMs)/Large Language Models (LLMs) present tricky deployment challenges. The process of training and deploying these models is iterative*. This iterative nature of ML deployment, combined with the fact that real-time and batch inferencing each come with their own constraints, makes ML inference challenging. New to LLM inference? Start with: What is LLM Inference?

The addition of LLMs to the ecosystem of ML models has made things even more complicated and introduced new challenges.

*check Token 1.17: Deploying ML Model: Best practices feat. LLMs, where we discussed deployment in detail.

In today’s Token, we will cover:

This article will be valuable for both seasoned ML practitioners and non-specialists. Experts will appreciate learning the differences between ML and FM inference and its optimization, while non-specialists will benefit from clear explanations of how these technologies work and their importance in the ML landscape.

LLMs are extremely sophisticated models compared to conventional ML models. The table below shows the number of trainable parameters in some widely used LLMs.

Compared to more traditional Convolutional Neural Networks (CNNs) models like AlexNet (62M parameters), ResNet-18 (11M parameters) and Inception V3 (23M parameters), LLMs are more complex in terms of their architecture and the volume of data they are trained on. More parameters mean the model has more weights to adjust during training, which requires more memory and processing power. This is true for both CNNs when dealing with image tasks and LLMs for language tasks. But LLMs, due to their enormous size, entail higher computational costs for training and inference, often requiring specialized hardware like GPUs or TPUs and substantial electrical power, to process large LLMs efficiently. (check this deep dive into AI chips). This increases the cost and complexity of deploying these models for real-time applications, making it a significant consideration for practical use.

LLMs require substantial memory for model weights, the KV cache, activations, and runtime overhead. Falcon 7B needs about 14 GB just to store FP16 weights, so 16 GB of RAM is a practical minimum for a basic CPU-loaded setup; quantized versions can fit in less memory. The exact requirement changes with precision, context length, batch size, and the serving framework.

ResNet, one of the most advanced CNNs, can be loaded using <6 GB of RAM. On the contrary, models like GPT 3 and BLOOM require 350GB+ memory.

There is a bright side though: while memory constraints present challenges for LLM inference, ongoing advancements in optimization techniques, hardware, and cloud computing are helping to mitigate these issues. Understanding and leveraging these developments is crucial for effectively deploying and utilizing LLMs and other memory-intensive models.

The tooling is no longer immature; it is specialized. vLLM and SGLang focus on high-throughput serving, TensorRT-LLM is optimized for NVIDIA GPUs, llama.cpp and Ollama cover local and edge use, and ONNX Runtime GenAI targets cross-platform and on-device execution. The right choice depends on the model family, hardware, latency target, and traffic pattern.

Modern engines expose many of the same optimization techniques – continuous batching, paged KV caches, prefix caching, quantization, speculative decoding, and distributed parallelism – but no engine wins on every workload. Let’s start with model serialization.

With a conventional model trained using libraries like scikit-learn, TensorFlow, and PyTorch, you could generate a pickle file* and use it for inference. To streamline the process and make it efficient in a team where engineers use a variety of libraries, you could use tools like ONNX to serialize the model such that it is invariant to the library that actually produced the model artifact.

A pickle file is a way to save Python objects, such as lists or models, so you can load and use them later.

Python’s pickle remains a poor format for production model exchange: loading untrusted pickle files can execute code, and the format does not solve portability across runtimes. Modern LLM deployments usually separate weights, tokenizer files, and configuration, using formats such as Safetensors, GGUF, or ONNX according to the target runtime.

ONNX Runtime GenAI now provides the generation loop around ONNX models, including token processing, sampling, KV-cache management, and constrained decoding. It supports CPU and several hardware execution providers, including CUDA, DirectML, OpenVINO, QNN, and TensorRT RTX.

Its model builder currently covers architecture families including DeepSeek, Gemma, Granite, Llama, Mistral, Nemotron, Phi, Qwen, SmolLM3, and Whisper, among others. That is much broader than the short list available when this article first appeared, although support should still be checked against the exact checkpoint and hardware backend.

Besides, AWS Sagemaker also allows users to deploy popular models for text and image generation in a couple of steps.

Serialization is only one layer. In production, the serving engine determines how requests are scheduled, how the KV cache is managed, and how efficiently the hardware stays busy.

Do not choose between them from a single vendor benchmark. Test the exact model, prompt-length distribution, output length, concurrency, and service-level objective. The fastest engine for offline throughput may not deliver the best time to first token for an interactive product.

GPUs usually outperform CPUs for large models and concurrent workloads because they provide much higher memory bandwidth and parallel compute. That does not make a GPU the cheapest option for every service: a small quantized model with sparse traffic may run economically on a CPU, while latency-sensitive or high-throughput workloads usually benefit from an accelerator. Measure the real request mix before deciding.

Here comes TensorRT. It is a runtime optimization toolkit for deep-learning inference on NVIDIA hardware. TensorRT turns a trained network and its weights into an optimized engine using graph optimization, layer and operation fusion, precision conversion, kernel selection, and memory planning.

NVIDIA provides related paths for different frameworks: Torch-TensorRT for PyTorch, TF-TRT for TensorFlow, and TensorRT-LLM for large language models. TensorRT-LLM adds LLM-specific scheduling, attention, KV-cache, quantization, streaming, and parallel-execution features on NVIDIA GPUs.

Overall, TensorRT plays a crucial role in optimizing LLM inference, enabling smoother deployment and improved performance for these complex models. For a broader look at how the inference hardware landscape is evolving –  from NVIDIA Vera Rubin to MatX and Taalas –  read our deep dive on the Inference Chip Wars.

Actually, you can be even more efficient by choosing the right hardware. ML inference can be done either in real-time or batch.

For real-time inference, a CPU can be economical for small quantized models and low request volume. As concurrency, context length, or model size grows, GPU memory bandwidth often becomes the limiting factor and an accelerator may reduce both latency and cost per token.

For large batch workloads, GPUs and TPUs are usually more efficient because matrix operations benefit from parallel compute and high memory bandwidth. CPUs are not automatically wrong, but they are rarely the first choice for sustained high-throughput generation. The graph below compares training runtime for an LSTM network; it illustrates hardware scaling, but it is not an LLM-inference benchmark and should not be used by itself to select production hardware.

LSTM, or Long Short-Term Memory, is a type of recurrent neural network designed to retain information across long sequences, such as language or time-series data.

Even though the experiments are performed using LSTM (for training), you will find a similar trend with other models including LLMs for inference when using a batch workload.

The graph illustrates how parallel hardware can scale with larger batches in one older training benchmark. It does not establish that a TPU is always optimal for LLM inference. The result depends on the model, numerical format, compiler and runtime, batch shape, memory capacity, and the service-level objective.

Operating GPUs and TPUs might be infeasible for you if you are an individual or small startup. Furthermore, you may lack the manpower to maintain this hardware. In such cases, you can opt for cloud providers that provide such accelerators. (Check this lecture from Cornell University about AI accelerators.) The most known examples are AWS Trainium and AWS Inferentia.

AWS Trainium is Amazon’s accelerator family for model training. AWS Inferentia is the corresponding family designed for inference; current generations are available through Amazon EC2 and AWS’s Neuron software stack.

AWS positions Inferentia around price-performance for deep-learning inference. As with any vendor claim, compare it using your own model, precision, prompt lengths, concurrency, and regional cloud pricing rather than treating a published benchmark as universal.

Other options include Google Cloud TPUs, NVIDIA and AMD GPU instances, AWS Trainium and Inferentia, and specialized inference providers such as Cerebras, Groq, SambaNova, and Together AI. Run:ai and DataRobot are infrastructure or platform software rather than accelerator chips themselves.

Finally, there are a few other tools that might ease the process of ML training and inference with LLMs. They are:

LLM inference remains difficult because memory, latency, throughput, and cost pull in different directions. The tooling is now substantially more mature: vLLM and SGLang cover high-throughput open-model serving, TensorRT-LLM optimizes NVIDIA deployments, ONNX Runtime GenAI extends execution across devices, and managed endpoints remove much of the operational burden. The challenge is no longer finding a tool; it is matching the engine, hardware, precision, and scheduling policy to the workload.

The durable workflow is simple: define the latency and cost target, benchmark a realistic workload, change one variable at a time, and recheck output quality. Quantization, batching, caching, speculative decoding, and parallelism can all help, but each introduces trade-offs that a synthetic headline benchmark can hide.

Thank you for reading, please feel free to share with your friends and colleagues. In the next couple of weeks, we are announcing our referral program 🤍

Previously in the FM/LLM series: