Much of the spotlight in AI falls on foundation models (FMs) and their subset – large language models (LLMs). Despite this trend, the cornerstone of machine learning success remains in deployment – turning these sophisticated algorithms into practical, operational tools. In systematizing the knowledge about the FMOps infrastructure, we want to highlight that whether it's a traditional ML model or an advanced LLM, the deployment process shares many similarities. There are some additional considerations, of course, but a lot of what you heard might just be hype ;)
The earlier you deploy the model, the earlier you become aware of production issues. As a result, you can tackle them before you are set to deploy the final version of your model. Today, we'll cut through the complexities, highlighting the key parallels and unique challenges in deploying both traditional and advanced models. This streamlined approach demystifies deployment, equipping you with best practices that you can follow even before training an ML model to make it easier and faster to deploy.
Model deployment means making a trained model available for real use, then keeping it reliable as requests, data, and requirements change. Serving predictions is only the beginning: the system also needs monitoring, access controls, versioning, and a way to roll back a bad release.
In today’s Token:
How to choose the right model?
Where should you store embeddings?
Do you need a feature store?
When should you add RAG?
How do you choose deployment infrastructure?
What should you monitor after launch?
Open-source deployment tools
Key takeaways
FAQ

Let’s get started with how you can choose the right model.
How to Choose an ML Model for Production
Well, the answer is, to use the simplest model that can get the job done. For instance, if you are working on a binary classification problem, start with logistic regression. There are two benefits to using the simplest model:
Simpler models are explainable
Simpler models can be trained quickly, and the team can focus on other phases of the ML lifecycle. This reduces the time to deployment.
For LLMs, this translates to using good enough models and prompt engineering instead of fine-tuning or training a custom LLM. For tasks like information extraction, where the required answer is present in the prompt itself, prompt engineering is very effective and less time-consuming than fine-tuning or training a custom model.
Choosing between open-source and closed-source foundation models involves weighing operational, financial, and strategic factors.
Closed-source models offer ease of use and robust support but come at a higher cost and with limitations in customization and potential provider dependency. They are ideal for those who prioritize reliability and simplicity over technical control.
Open-source models, while more affordable upfront, demand greater technical expertise and can incur hidden operational costs. They offer customization and innovation opportunities, but require significant investment in infrastructure and expertise management.
Where Should You Store LLM Embeddings?
Language models compute internal representations as they process tokens. The embeddings we store for search are a different part of the system: an embedding model turns documents and queries into vectors so we can find related content. But can we store these embeddings like a normal text or image file?
Why Use a Vector Database for Embeddings?
Embeddings can be stored in a file, but searching a growing collection usually calls for an index. A vector database can combine similarity search with filtering and updates. You do not always need a separate database: PostgreSQL with pgvector can also serve this purpose, and a small collection may need only an in-memory index.
Choose based on retrieval quality, query latency, filtering, update frequency, and the size of your collection. A faster index is useful only if it still finds the documents your application needs.
Do LLM Applications Need a Feature Store?
In conventional ML projects, feature stores help store and share features and facilitate collaboration among the data scientists and engineers involved in the project. How can LLMs take advantage of the feature store?
In numerous applications, input to the language models consists of features like user attributes, product attributes, and history in addition to the textual prompt and context. For example, for a book recommendation, to recommend the most suitable book, the language models will most probably need the attributes of the book, user attributes, and history in addition to the abstract of the book. A feature store can help when these attributes need consistent definitions, freshness, and reuse across models. It is not mandatory for every LLM application, and it does not have to be a separate store for embeddings.
When Should You Add RAG to Your LLM Application?
Getting the optimal performance from the model depends on the underlying task you are planning to use the model for. For a simple task, where the context in the prompt has the answer to the problem, you can get decent performance with prompt engineering.
When the task involves finding answers in a changing or private knowledge base, consider Retrieval Augmented Generation (RAG). It retrieves relevant material and adds it to the model’s context before generation. RAG works alongside prompt engineering, not instead of it. Check retrieval quality and whether the answer is actually supported by the sources; retrieval alone does not prevent hallucinations.

Image Credit: RAG for Knowledge-Intensive NLP Tasks
ML Model Deployment Infrastructure: Cloud vs. On-Premise
For the deployment of ML models, you have three major options:
Deploy in your server
Deploy in the public cloud
Hybrid deployment (combination of private resources and public cloud)
If you choose the first option, you will be responsible for setting up the infrastructure, allocating the resources, and maintaining the server. This will also require a skilled system administrator in the team and some heavy upfront costs. If you have the resources to deploy the ML model in your server, you will enjoy benefits like:
Fine-grained control over the resources
Control over data location and security configuration, with responsibility for securing the system
Potential long-term savings when utilization justifies the hardware and operating costs
If you don’t want to go through the hassle of setting up your own infrastructure, you can choose to deploy the model in public clouds like AWS and Azure. The pros to deploying your model in the public cloud include:
Low upfront cost and easy scalability (public clouds provide auto-scaling which allows automatic instantiation and termination of hardware resources depending upon the usage)
Services like Amazon SageMaker AI provide managed training, deployment, and other ML services. Compute, storage, and other billable resources still carry costs.
Managed infrastructure does not remove your responsibility for security. Check service eligibility, contracts, access controls, encryption, and data handling against your organization’s requirements.
Finally, we can combine private resources with public cloud services. For example, sensitive requests can stay on-premises while approved workloads run in the cloud. But preprocessing data does not automatically make it safe to send elsewhere: embeddings can still contain sensitive information. Decide what may cross the boundary before choosing the architecture, and enforce that decision through access controls, routing, and logging.
Self-hosting also means owning capacity planning, security updates, and failures. For a small or unpredictable workload, compare that operating burden with a managed API before buying hardware. For either approach, test overload, timeouts, and recovery—not just a successful demo.
Today, we have several hardware options for training and deployment: CPUs, GPUs, and specialized accelerators such as TPUs. CPUs do support parallel execution and can be a practical choice for smaller models or modest traffic. Large LLMs often need accelerators for inference as well as training. Choose based on model size, available memory, latency, throughput, and cost—not a rule that training belongs on GPUs and inference on CPUs.
Regardless of the hardware used for deployment, there are a few techniques that can help you improve the inference speed:
Quantization: Represent weights, and sometimes activations or the KV cache, at lower precision—for example, INT8 or INT4. This reduces memory requirements; speed gains depend on the hardware and serving engine. Test quality after the change.
Pruning: Remove weights or larger structures from a network. Fewer nonzero weights do not automatically mean faster inference: the runtime and hardware must be able to exploit the resulting sparsity.
For LLM serving, continuous batching and prefix caching can also help. Speculative decoding is another option, but its benefit depends on the model and workload. Measure time to first token, time per output token, throughput, tail latency, and cost on representative requests.
ML Model Monitoring in Production
Machine learning is an iterative process and doesn’t finish after deployment. A model can keep producing fluent answers while the product catalog, customer behavior, or underlying data has changed. Monitor both the service and the quality of its predictions. Updating a retrieval source may be enough in one case; changing the model may be necessary in another. Either way, evaluate the change before rolling it out.
Data and Concept Drift:
Data drift refers to the change in data distribution with time. As time passes, people’s preferences change. For instance, a user who likes to watch romantic movies today may prefer watching action or suspense after a year or so. As seasons change, people’s choices for their drinks change (beer in summer and whisky or rum in winter). A change in inputs is a signal to investigate, not proof that the model has failed.Concept drift means the relationship between inputs and the outcome changes over time. A common example of concept drift is the behavior of people during the COVID lockdown. Before the lockdown, for a restaurant or its mobile app, weekends would be the right time to send offers but during COVID the behavior changed and people wouldn’t respond to those offers.
Since, language models are used in conjunction with user/product data (Customer Support bots, Personalization/recommendation services), they are equally impacted by data and concept drift.Respond to measured degradation: check the data pipeline, update sources or prompts where appropriate, and retrain only when evaluation supports it. Keep a tested rollback path.
Resource usage, errors, and bugs:
Once the model is deployed, apart from issues related to the accuracy of the machine learning model, it may suffer from issues like disproportionate resource usage, faults and errors, fluctuating user requests, network congestion, etc. Only what gets measured, gets managed, meaning we can tackle the problem only after identifying it. You can use loggers to log these issues and tools like Grafana to visualize the metrics like latency, request volume, error rates, etc. Once you identify the problem, you can tackle them using the right approach i.e load balancers to uniformly balance the load on the resources, auto-scaling to automatically scale up or down the active resources, etc.Model-quality metrics are a separate layer: see what to measure in LLM monitoring.
Compliance and Regulations:
Depending upon the quality of input data the model is trained on, the model may generate outcomes that are morally incorrect or fail to comply with regulations. For instance, a few years back, Amazon built an AI tool to find the best candidate for the job. Later, upon inspection, it was found that the AI was not gender-neutral. It penalized resumes that contained the word “women”. Even the most advanced language models aren’t immune to such problems, GPT-3 was found to capture anti-muslim bias.Guardrail models can help, but they are not a guarantee. During data preparation, classifiers can flag material for review. During deployment, guardrails can screen inputs and outputs or escalate uncertain cases. Test false positives, missed violations, and performance across groups; filtering does not by itself establish fairness or compliance.
Open Source Model Deployment Tools
TensorFlow Serving (github)
A flexible, high-performance serving system for machine learning models, designed for production environments. It's part of the TensorFlow Extended (TFX) ecosystem.
Not specifically for foundation models, but can handle various types of TensorFlow models.
TorchServe (github)
Developed by AWS and Meta, TorchServe is a model-serving framework for PyTorch. Its repository was archived in August 2025.
The maintainers state that no updates or security patches are planned. It remains relevant to existing deployments, but is not a default choice for a new one.
MLflow (github)
An open-source platform for managing the end-to-end machine learning lifecycle. It includes capabilities for deploying models in diverse environments.
General purpose, not limited to any specific model type.
Kubeflow (github)
A Kubernetes-based platform to deploy, monitor, and manage machine learning models at scale.
Not specific to foundation models but highly scalable for various model types.
Seldon Core (github)
An open-source platform that enables deployment, scaling, and monitoring of machine learning models in Kubernetes.
Versatile for different model types, including large models.
For LLM inference specifically, the serving engine is another choice. These tools do different jobs: KServe manages serving on Kubernetes, while engines such as vLLM and SGLang execute model requests.
Tool | Primary role | Best fit |
|---|---|---|
High-throughput model serving | OpenAI-compatible GPU inference | |
TensorRT-LLM | NVIDIA inference optimization | Optimization for supported NVIDIA hardware |
llama.cpp | Local and edge inference | CPU, Apple silicon and quantized models |
Production text-generation server | Existing deployments; now in maintenance mode | |
KServe | Kubernetes model serving | Platform teams standardizing many models |
Model serving with prefix caching and continuous batching | LLM workloads with reusable prefixes; benchmark against vLLM on your own traffic |
LLM Deployment Best Practices: Key Takeaways
Deploying an ML model is an iterative process. Start with the simplest model that can get the job done, choose infrastructure that fits your data and workload, and measure what happens under realistic traffic. A vector index or feature store can help where the application needs one; neither is mandatory for every model.
Quantization, batching, and caching can reduce serving costs, but check their effect on quality and latency. After launch, monitor the system, investigate changes, and evaluate updates before release. Keep a way to return to a version you trust.
FAQ
Why do ML models fail to reach production?
Common causes include unclear ownership, weak data pipelines, no business success metric, integration cost, latency or security gaps, brittle evaluation, and no plan for monitoring and maintenance.
How do guardrail models prevent biased LLM output?
Guardrail models classify or judge inputs and outputs against defined policies. They can block, route, redact, or escalate content, but must be tested for false positives, false negatives, and group disparities.
Should you fine-tune an LLM or use prompt engineering?
Start with prompting and retrieval for fast iteration. Fine-tune when you have representative data and need repeatable behavior, format, style, or task specialization that prompting alone cannot deliver.
What is the difference between data drift and concept drift?
Data drift means the input distribution changes. Concept drift means the relationship between inputs and desired outputs changes, so yesterday’s mapping is no longer valid.
What is the best way to deploy an LLM?
Choose the simplest architecture that meets quality, privacy, latency, scale, reliability and cost requirements. Benchmark managed APIs and self-hosted options under realistic load before committing.
Please give us feedback
Thank you for reading, please feel free to share with your friends and colleagues.







