What are Guardian Models?

One of the main reasons to be cautious about AI is the range of harms it can produce – from malicious use in attacks, fraud, or disinformation to harmful errors and hallucinations in benign responses. Just as we are not fully protected from criminals in everyday life, neither is AI. The solution is an added safety layer: specialized guardian models trained to detect and filter unsafe prompts and outputs. These “AI soldiers” are designed to make the ecosystem more secure and reliable.

In 2025, guardian models aren’t niche experiments – they’re built into every serious AI deployment. OpenAI has its moderation layers, Microsoft offers Azure Content Safety, Meta has shipped Llama Guard since 2023, IBM provides Granite Guardian, and startups are releasing their own open-source versions. So guardian models are quietly powering almost every chatbot or generative AI you use – without most people even realizing it.

We don’t talk much about AI safety in our AI 101 series, but today is finally the day for it. We are going to dive into the basic definitions of guardian models (since there can be some uncertainties), explore the main guardian models used today, and look at the newest dynamic approach, DynaGuard, that can enforce any set of rules you give it at runtime.

Guardian models may seem similar to typical LLMs, but that’s only at first glance. And you definitely need to know about them to save your models and protect yourself.

In today’s episode, we will cover:

The main rule goal for guardian models is to effectively enforce content policies and rules in real time. They run alongside the primary model, monitoring inputs and outputs in real time to catch harmful or policy-violating content. So you don’t need to hard-code every rule in the primary model.

In one of our interviews, Amr Awadallah, founder and CEO of Vectara, emphasized the importance of a separate safety and verification component that oversees the model – a guardian agent, that monitors LLM outputs to catch hallucinations and triggers human-in-the-loop when risk is high.

This “AI supervising AI” loop is a cool shift, as it turns models into generators and regulators at the same time.

Guardian models also don’t need to be massive-scale systems. They are usually comparatively small (2B-8B parameters), but still manage to reliably catch harmful content across dozens of risk categories.

Now to the differences in terms.

You’ve heard “guardrails” and you might have heard “guardian model.” Are they the same? They’re closely related and often used interchangeably, since both enforce safety rules on AI systems. But in practice, a guardian model is one way to implement guardrails, while “guardrail” is the broader concept. Guardrails usually mean the full toolkit of measures – rules, filters, or models – that keep AI behavior in check.

Importantly, guardian models go beyond simple filtering. They can:

RAG hallucination risks appear because RAG can still “hallucinate” if the retrieved context is irrelevant or conflicting.

Some guardian models like Llama Guard and ShieldGemma control only harmful content, while others, IBM's Granite Guard, for example, mitigate RAG issues as well. Let’s dig into the tech specs of these AI “guards” to see how they’re built and how they really work →

Let’s start with Llama. In 2023, GenAI at Meta firstly introduced Llama Guard, an LLM-based safeguard model, that uses a safety taxonomy – a set of categories covering legal, policy, and safety risks – and can classify user prompts and AI responses. It was instruction-tuned, and could be adapted to new taxonomies with zero-shot or few-shot prompting. Then in 2024, they developed Llama Guard 3 Vision – a multimodal guardian model with expanded image understanding capabilities.

Now things get more interesting with Meta’s Llama Guard 4, its latest safety system that works with both text and images. It checks whether user prompts or model outputs are safe or unsafe. If unsafe, it also points out which rule was broken, for example, violence, hate, privacy, intellectual property, etc.).

It has 12 billion parameters and is based on Llama 4 Scout, but pruned and fine-tuned specifically for safety classification. Llama 4 Scout is originally a Mixture-of-Experts model.

For Llama Guard 4, Meta pruned and kept only the shared expert of Llama 4 Scout, resulting in a dense feedforward early-fusion architecture for efficiency. Early-fusion means that the both image and text inputs are merged together before going through most of the transformer layers. So Llama Guard 4 is lighter than Scout version while still retaining knowledge. It is also small enough to run on a single GPU.

Unlike earlier versions, Llama Guard 4 can evaluate prompts that include multiple images as well as multilingual text. Compared to the previous Llama Guard 3 it has the following results:

(Recall defines how much harmful content that actually exist the model caught. F1 score is the metric that merges recall and precision, showing how good a model is at catching harmful content while not over-flagging safe content.)

However, Meta’s guardian model is not a solution for everything. It can suffer from the following limitations:

Another model, similar to Llama Guard, is Google’s ShieldGemma, a suite of safety content moderation models, firstly introduced in 2024. Built on Gemma 2, it was designed to catch harmful text in user prompts and AI responses. Now with the release of Gemma 3, Google has launched ShieldGemma 2, a 4-billion-parameter guardian model, that extends safety checks beyond text into synthetic and real-world images. For a deeper look at the Gemma 4 architecture that powers the latest generation of these models, see our Gemma 4 breakdown

For engineers, a few specs stand out. The model is compact enough for efficient deployment (4B parameters) yet robust across policy domains. It supports both input filtering (screening images before they enter vision-language pipelines) and output filtering (blocking unsafe generations). Google has released the weights under an open license, with support for custom policy tuning, so teams can adapt the classifier to their own risk frameworks. The training was optimized on TPUv5e hardware, and inference latency is kept competitive for real-time moderation.

Developers can adapt classifications to their needs. For example, probability thresholds can be set higher or lower depending on whether ShieldGemma 2 is filtering a dataset, guarding an image generator, or supporting a multimodal chatbot. As it is built on Gemma 3 (it’s utterly confusing that ShiledGemma 2 is built on Gemma 3), it is available through major AI frameworks such as Hugging Face Transformers, Keras, and JAX, with community support also extending to Ollama.

Like other safety systems, ShieldGemma 2 has some limits – it may struggle with very subtle or culturally specific harms, or with image styles it wasn’t trained on.

To sum up, Llama Guard and ShieldGemma focus on detecting harmful content across multiple dimensions. But it is only one part of the problem. The next model that we cover emphasizes the importance of broader capabilities for guardians →

Granite Guardian from IBM stands out as it goes beyond surface-level problems deeper into the model issues. It combines several categories of risks into a single, more general system:

Granite Guardian was trained on about 7,000 human-annotated prompt–response pairs from HH-RLHF, each triple-labeled across safety and specific risks, such as bias, jailbreaking, violence, profanity, sexual content, unethical behavior, AI refusal. Additional low-confidence cases were targeted to strengthen coverage. Synthetic datasets expanded training with:

Granite Guardian has an interesting training process:

As a result, Granite-Guardian-3.2-5B scored:

But where does IBM’s model falls short?

Overall, Granite Guardian provides a more complete safeguard: it can be used as guardrails for moderation, as evaluators for quality checking, or as filters in RAG pipelines and agentic workflows, making AI safer and more reliable across different applications. Being open-source, it also supports these values across the wider community.

As we have seen, typically, guardian models work with fixed categories of risk and the problem of context sensitivity remains a challenge. Sometimes users need to define the unwanted content by themselves. For this, more dynamic and flexible guardian models are needed.

In some cases, what is considered as “bad” content depends on context. Take retrieval models, for example: A chatbot should not help plan violence, but it should be allowed to summarize violent events from the news. Static harm categories can miss a lot of real-world problems.

To emphasize the importance of specific rules, researchers from University of Maryland and Capital One developed DynaGuard, a new guardian model designed specifically to handle custom rules.

Here is what makes DynaGuard stand out compared to other guardians:

So how does this new approach work?

The main thing for DynaGuard is training, as for other guardian models. For this stage researchers created special dataset – DynaBench. It consists of 40,000 unique policies for training. Here is how it was build:

DynaGuard has notable performance gains:

However, DynaGuard’s reliance on explanations raises open questions about how best to integrate them into multi-agent recovery pipelines. Also, it is unclear how to be with ongoing updates as new use cases emerge.

Anyway, DynaGuard is a great step to the customization of guardian models that will work for the particular user. It makes safety modular and programmable, almost like a firewall where you update rules on the fly.

In addition to open models that we have mention, many leading AI platforms use dedicated moderation or guardrail systems behind the scenes.

Also, here are a couple of classical and quiet new guardian systems:

This is what happens today in a landscape of guardian models. These lightweight models do their quiet job, paving the way to trustworthy AI. It is a good strategy to relieve the main model of this work and entrust it to a specialized one. Unseen and maybe a little bit forgotten, guardian models keep control of all the malicious content we could encounter (not with 100% efficiency, but actually there is no ideal model invented yet in any AI field). After all, approaches like DynaGuard demonstrate the direction toward more customizable rules on what is harmful and what is not, highlighting that guardian models should adapt better to new rules. Other systems that we explored today show that guardian models are everywhere – from multimodal models to agentic systems and MCP. And we are very curious about what the next step for trustworthy and safe AI will be.

Sources and further reading

Resources from Turing Post