PAN World Model Explained: Physical, Agentic, Nested

We have devoted a couple of articles to exploring the bottomless world of world models. Firstly, we looked at an entire ecosystem – NVIDIA’s Cosmos World Foundation Model platform that supports the path toward Physical AI, and then we dived deeper into what current world models look like and how they function to bring simulated worlds into the service of AI agents. But there is so much more to say about the field that is still in its formative stage.

Today let’s fundamentally rethink how we build world models. Maybe the way we usually approach their architectures is not the right one? When the field is just establishing, there is no completely right or wrong, there is a world full of investigation. Since there’s still disagreement about what a world model even is, figuring out how to build one is even harder.

Researchers from Carnegie Mellon University, MBZUAI, and UC San Diego have taken a critical path through world modeling. Starting with the imaginative worlds of Dune and the inspiration from the concept of “hypothetical thinking” in psychology, they identify the main mission of world models: “Simulating all actionable possibilities of the real world for purposeful reasoning and acting.” They argue that a new architecture should be:

Hierarchical and multi-level
With mixed continuous and discrete representations
Generative and self-supervised

Such an approach, as the researchers see it, could lead to a PAN world model system, meaning Physical, Agentic, and Nested, that might form the foundation of AGI. So let’s follow their critique experience to explore the deeper sides of effective world modeling that could shape our future.

In today’s episode, we will cover:

Why do we need world models?
The pitfalls of current world models
Inspiration and new ideas come in
How to build a better world model
How does PAN world model work?
The main advantages of PAN
Not without limitations
Conclusion
Sources and further reading

Why do we need world models?

We will start with a quick reminder of what a world model is (say, how most see what it is) and why we really need this kind of tools.

A world model comes into play when we think about agents – systems that act on their own to achieve goals. Agent’s environment is everything around it: the physical world, the social world, or even the universe itself. Generally, an optimal agent works like this:

It sees the current state of the world.
Based on that, it picks an action following a certain strategy or rule.
The environment responds, producing a new state.
The agent also receives rewards that measure how well it’s doing toward its goal. Its aim is to pick actions that maximize the expected total reward.

In theory, an optimal agent would need access to the true state of the world in order to make perfect decisions, but in practice, it has imperfect information with noise and incompleteness. World models can help with this giving agents some kind of “imagination”.

A world model functions like a generative simulator of futures, allowing the agent to do “thought experiments” before acting in the real world.

With a world model things become easier avoiding the need of the true state for agents. Instead, we observe the following process:

Image Credit: Critiques of World Models original paper

The agent builds its own belief state (ŝₜ), based on what it observes (oₜ) through its sensors.
An encoder processes these observations into that internal belief state.
Then the agent considers a possible action (a′ₜ).
The world model predicts what the next belief state (ŝₜ₊₁) of the world will look like not deterministically but probabilistically, as there are usually many possible futures and uncertainties.
The agent repeats this prediction–action cycle over multiple steps into the future.

With this ability, an AI agent can run thought experiments, like “If I do action A, what’s likely to happen?”, “What if I try B instead?”, “What if I do nothing?”, etc. This allows the agent to choose actions that are most likely to lead to its goals.

A general-purpose world model can cover many domains, such as:

Physical dynamics (how objects move, how water pours, etc.)
Embodied experiences like balance, posture, or how it feels to be hot or dizzy.
Emotional states
Social situations
Mental world: abstract reasoning like planning, strategizing, or problem-solving in multi-agent situations.
Counterfactuals – exploring “what if” scenarios.
Evolutionary dynamics like changes over generations, such as inheritance and adaptation.

The strength of a world model is that it allows simulative reasoning: running through many imagined futures, comparing them, and picking the best plan. Moreover, world models make it possible for AI to transfer knowledge from one setting to another, like humans do with different skills (for example, a gamer’s skill at controlling virtual characters helps with drone piloting). This is a way to create machines that could achieve cool level of “zero-shot generalization” in new situations without explicit training.

And what connects all world models is the bold idea: If models can predict the next word, we can make them predict the next “world”, like every possible way the future might unfold. And the main question here is how to do this.

The pitfalls of current world models (and PAN vs Nvidia Cosmos)

The first thing that catches the eye is that most today’s world models share a strong emphasis on video and image generation. That’s why they often look more like video generators than true reasoning engines. There is something more to consider in different world models that we work with today:

Gaming world models like Genie 2 (Google DeepMind) and Muse (Microsoft) are good for short visual simulations lasting 1-2 minutes, but they struggle with long-term coherence and are too domain-specific for console-style inputs or Minecraft-like worlds.
3D scene world models like those built by World Labs push realism in space with the focus on stylized 3D environments and navigation, but they don’t yet support agent reasoning or causal simulation and lack dynamics, physics, or rich interactivity.
Physical world models, such as NVIDIA Cosmos and Wayve GAIA-2 designed for embodied physical tasks specialize in modeling physics and sensory-motor responses. However, they don’t generalize to open-ended, multi-agent, or social reasoning tasks, staying capable only in narrow areas.
Video generation models like OpenAI Sora, Google DeepMind Veo that create high-quality video sequences from prompts or prior frames only generate fixed video trajectories and have no explicit state, action, or object-level understanding.
And finally, Joint Embedding Predictive Models, Joint Embedding Predictive Architecture (JEPA) with V-JEPA, DINO-WM and others, are the most promising direction with the ability to, for example, perform real robotic arm manipulation, but they are still far from handling complex, long-term tasks.

So there is still no completely winning architecture for world models that would offer us everything we want to see from such systems. Maybe it is time to reconsider what we can do with this? One practical answer came from Princeton and UC — Web World Models combine deterministic web code with LLM generation to build controllable, open-ended environments for AI agents, addressing many of the structural gaps described above.

What does PAN mean? Physical, Agentic, Nested

With the aim of creating a new architecture that might solve a lot of limitations, the researchers from Carnegie Mellon University, MBZUAI, and UC San Diego turned for inspiration to two concepts.

The first one is what we can see in futuristic stories. →

The world models’ idea shows up in science fiction, most famously in Dune. In this story, the Kwisatz Haderach is a figure who can recall the memories of ancestors and foresee countless possible futures, choosing the path that leads to the best outcome. That power allows them to command armies, transform societies, and overall reshape the fate of whole planets.

On the other side, we have human brain, a universal system handling a huge variety of tasks, from simple and physical to highly abstract. Each task can be seen as a challenge in a multimodal environment – one that mixes physical, social, emotional, and cognitive factors, spread over both short and long timescales. What helps humans cope this is hypothetical thinking from psychology, or what we might casually call “what if” thinking: mentally simulating possible futures and testing different outcomes before acting. This is exactly what a world model aims to capture.

So the researchers argue for a new design they call PAN (Physical, Agentic, Nested). Their idea is to build a world model that:

Uses data from all types of experience: text, images, actions, etc.
Combines continuous and discrete representations for smooth processes and clear categories.
Works through hierarchical generative modeling, enhanced by LLM backbones.
Trains with a generative loss tied to real-world observations.
Improves agents through reinforcement learning on simulated experiences.

Before we draw any conclusions on the effectiveness of PAN, let’s see how it is built and how it works.

How to build a better world model

To build something new you need to reconsider what you already have, right? The researchers took a path of criticizing patterns that have been proposed by big voices like Yann LeCun. They are influential, but at the same time controversial as the community keeps exploring the area. Well, here are these basic patterns for building world models across five aspects:

Data → Sensory data, like video, audio, smell, touch, is superior to text because the real world generates vastly more sensory data.
Representation → World states should be continuous embeddings, not discrete tokens, because this allow for easier optimization with gradients, which is how most machine learning is trained.
Architecture → Autoregressive, generative models are doomed, since they accumulate errors and can’t handle uncertainty.
Objective → Probabilistic encoder–decoder (data-reconstruction) objectives don’t work. Instead, predict directly in latent space, using latent reconstruction loss.
Usage → World models should be used with Model Predictive Control (MPC), not Reinforcement Learning (RL). MPC requires fewer training trials, making it more sample-efficient and safer.

As these points are controversial, this means they are not the absolute truths, so why not to critique them? This is exactly what the researchers behind the PAN world model did, creating their new building concepts for world models:

Data → Raw sensory data is huge in volume but often repetitive and low in meaning. Text, on the other hand, is a compressed form of human knowledge, representing abstract concepts, like justice, regret, social norms, that you can’t observe directly with sensors. A true world model should use all modalities (text, images, audio, etc.), because each provides unique insights.
Representation → Continuous data is flexible but also noisy and unstable. Humans don’t think in raw continuous signals, we also categorize them into discrete concepts like words and symbols as models do. A common concern with discrete tokens is that they may lose detail, but there is a solution – scaling representation capacity:
- by making the vocabulary bigger (scaling up), or
- by allowing longer token sequences (scaling out).
Scaling up grows vocabulary size exponentially with complexity, while scaling out grows sequence length logarithmically which is much more efficient. So with enough sequence length, they can represent complexity well. This means that with enough sequence length, discrete tokens can represent complexity well, approximating even very fine continuous differences.
Image Credit: Critiques of World Models original paper
However, the most promising direction is a hybrid approach where:
- Continuous embeddings capture fine-grained sensory nuance.
- Discrete tokens represent concepts, logic, and memory.
Architecture → Autoregressive models aren’t bad actually. For example, even JEPA is still autoregressive in practice, because it recursively predicts future states step by step. In general, skipping observation reconstruction risks unguided, unstable predictions.
Researchers proposed that a better approach may be hierarchical abstraction with generative latent prediction (GLP) architecture that has layered design:
Image Credit: JEPA vs GLP, Critiques of World Models original paper
- Low-level: Next-embedding predictors (diffusion models) handle noisy, fine-grained variation in perceptual data, for example pixels, audio, body sensors. They use encoder–decoder structures, grounding predictions in real data.
- Mid-level: Next-token predictors (autoregressive Transformers) operate on discrete tokens to capture symbolic structure, such as objects and actions, and compositional reasoning.
- High-level: An LLM system reasons over language tokens for long-term planning, abstract reasoning and counterfactuals.
Objective → Latent-only loss is fragile, that’s why latent predictions need real-world grounding, which generative reconstruction (predicting the actual next observation) can provide.
Usage → MPC is really useful, but it is computationally expensive and usually “looks” only a few steps ahead, which limits its ability to handle long-term reasoning. On the contrary, RL with a world model can train an agent policy offline through imagined rollouts, producing a reusable decision-making system that acts quickly at runtime.
Image Credit: Critiques of World Models original paper

Overall, a robust world model needs to combine strengths across modalities, representations, and training methods – this is all about more hybrid approaches. From these starting points the researchers from finally could build their novel architecture. →

How does PAN world model work?

First of all, to support the idea of the complexity and breadth of a world model’s action, PAN is motivated by a realistic, complex scenario – a mountaineering expedition.

High-level tasks include choosing gear, planning routes, assessing weather, coordinating with teammates, while low-level tasks are climbing, using ropes, adjusting posture, motor control under stress. Multimodal inputs combine sights (snow, cliffs), sounds (a teammate shouting), tactile sensations (wind, cold), body strain.

Here is how PAN world model works step-by-step, using hierarchical encoder – world model – decoder pipeline:

Image Credit: PAN-World architecture, Critiques of World Models original paper

PAN ingests the multimodal inputs 𝑜 with a Sensory Encoder (h) that splits into two pathways:
- Discrete pathway:
  A tokenizer converts signals into tokens at multiple abstraction levels. These can be:
  - learned discrete tokens via Vector Quantized Variational Autoencoder, VQ-VAE, or
  - natural language words.
  Tokens represent concepts like “where am I?”, “who is with me?”, “what tool do I have?”.
- Continuous pathway:
  In this case, low-level sensory details are encoded into continuous embeddings for perceptual nuance.
Together, these form the belief state.
Given an action, for example “buckle the carabiner”, PAN predicts the next world state using:
- An LLM-based backbone for symbolic reasoning with both natural language tokens and learned vocabularies.
- A diffusion-based predictor for low-level embodied reasoning.
A Learned Switch combines these depending on task demands.
Finally, a Multimodal Decoder (g) reconstructs the next observation and compares it with real data. It is also important that the decoder can output not just video, but sound, temperature, proprioception, or even text, grounding predictions in full sensory reality.

To train the PAN system researchers also follow specific strategy:

Each module is pretrained separately, like LLM on text, diffusion on video, etc., using self-supervision.
Then they are integrated using multimodal data, cascading embeddings, and gradient propagation.
Continuous modules use gradient optimization, while discrete ones may use gradient-free methods like RL.

Notably, PAN doesn’t need complete datasets of every complex activity, leveraging stratified data and learning from incomplete sources: books and maps for navigation, indoor climbing videos for motor skills, language data for social reasoning, and simulation data for embodied skills – this is a really huge variety.

Combining all these features, we have a system that moves from video generators to an internal sandbox for simulating futures.

The main advantages of PAN

Firstly, PAN stands out because it is a general-purpose world model.
It gets strengths from multimodal data (text, images, audio, sensory signals), hybrid representations (continuous + discrete), hierarchical generative models, and objectives grounded in real observations.
It demonstrates advanced agentic behavior. A PAN-Agent can generate possible world states and actions ahead of time, cache them for quick retrieval, and use them for deliberate planning or flexible reaction.

Image Credit: Critiques of World Models original paper

Combining RL and MPC, PAN enables agents to plan strategically, reason counterfactually, and also act with foresight and adaptability.

However, as usual →

PAN limitations and open challenges

PAN world model is still a proposal, not a proven system.
Its layers and complex architecture can cause engineering difficulty, high compute costs, and training instability.
Collecting comprehensive real-world data is quite unrealistic.
There are risks connected to RL. Agents may exploit flaws in the simulated world rather than learning robust strategies.

However, the main idea of PAN now is to rethink how to better build world models, and it is the future task to show what it really can do.

Conclusion

PAN is a promise for making world models general-purpose and truly actionable and adaptable. Its design follows five principles, that you can utilize in you developer’s experience:

Use all modalities of data.
Combine continuous and discrete representations.
Build a hierarchical generative model with two reasoning engines – an extended LLM backbone and a generative embedding predictor.
Train with generative loss grounded in observation data, not just latent loss.
Apply the model to simulate experiences for training agents with reinforcement learning.

This can make agents closer to our, human like cognition with navigating uncertainty, coordinating socially, and reasoning across time scales. As world models become the foundation for reasoning, imagination, and action, approaches like PAN with their grounding in experience, layered abstractions, and scalability can unlock more general capabilities than current world models.

The PAN framework is also an important development that highlights several future (and even current) directions:

Expanding from single-agent to multi-agent simulations, that are more interesting and comprehensive.
Covering different time scales, from milliseconds to millennia.
Increasing fidelity across modalities with better integration of vision, sound, and motion.
And finally, allowing agents to learn directly from imagined experiences.

We are very curious about the future of systems like PAN and other world models, and we hope that through a variety of approaches, researchers will lead us to a better world with truly comprehensive world models.

Sources and further reading