∇ Guide: How World Models Work – From Dreamer and JEPA to Waymo and Atlas

TL;DR: World models can be grouped by what they carry forward: generated observations, compact latent states, or explicit structures of objects and space. Observation-generating models are best for visual simulation, latent models for efficient prediction and planning, and structured models when geometry, objects, and physical relationships matter most.

AI is making its way into our workflows through text, code, images, and increasingly audio and video. But in the physical world, in factories or self-driving cars, even advanced reasoning models and personal agents aren’t enough. Their weak spot is reliably predicting what happens in the physical world.

World models try to capture how the world changes, including what might happen when an agent takes an action. That could help a robot predict whether pushing an object will move it or tip it over.

But there’s no universally accepted definition. The term covers different approaches, from simulators and video generators to representation models like JEPA. We just came back from a world model workshop, and every participant had their own definition. Several studies have tried to organize the field (we’ll link them at the end), but the categories widely overlap.

So how are these models built? We’ll move from Dreamer 4 and JEPA to Genie 3, NVIDIA’s Cosmos-Predict2.5, and World Labs’ Atlas, then look at what these ideas become in practice: Waymo’s world model for autonomous-driving simulation, agents that practice inside Minecraft, and object-centric models for robotics. Our goal is to explain the architectures in a way that’s easy to grasp and use, so you can judge what each can do, where it falls short, and which might suit what you’re building.

In today’s episode:

Before we jump into the classification of world models, let’s start with the overall patterns and functions.

World models interact with the world and its rules. The world never stays in one state for long. It is a sequence of states that change over time. For example, if we look from the side of mechanics, objects are at rest or move, and we can imagine this process as snapshots going one after another, every second, for instance. At each step, there will be a new state of an object. Then, from the side of science, we have chemicals that react and transform their state, cells that also change, and living systems responding to their surroundings. This is the varying physical world. But in a digital world, we also witness a number of states following each other – like how a system changes before and after running the code or with each line of it. A world model tries to learn the rules behind these transitions: not only what the world looks like now, but also what can happen next and why.

Depending on the environment, these rules may come from physics, chemistry, biology, human behavior, coding or all of them at once. But the general workflow stays the same:

In other words, a world model tries to capture and predict world dynamics. It compresses large amounts of visual and physical experience (often buried in pixels) into a smaller, useful understanding of objects, movement, cause and effect, and physical constraints. And another hard task, it needs to keep the important physical patterns while ignoring details like lighting, texture, and background noise.

As Dr. Fei-Fei Li described in her post on Substack, world models have three main functions:

Simulation connects the other two. If a model understands an object’s shape, physical properties, and response to force, it should be able to show that object from different angles, predict what happens when it is moved, and plan how a robot could interact with it.

To make these tasks easier to perform, the following properties play a huge role for world models:

Today, the main functions are usually handled by different world models. Renderers are already widely used, while simulators and planners are much less developed. The long-term goal is to combine all three into one model that can generate a world, simulate it accurately, and act inside it at the same time.

To see how we might get there, let’s unpack how different types of world models work.

A world model needs a way to represent the current situation and predict what can happen next. For an agent, that prediction also needs to account for its actions. There is no general taxonomy to classify world models because many architectural and workflow aspects often meet in one model.

Though, we propose to organize them based on the central architectural question – Reconstructive state-space: a generated observation, a compact latent state, or an explicit structure of objects and space? →

What does this all mean?

A generated observation – the model carries the world forward by generating what the next observation should look like, such as the next image, video frame, or sequence of tokens. It represents the future by recreating what an observer would see.

A compact latent state – instead of generating the full observation, the model compresses the important information about the current world into an internal representation and predicts how that representation will change. The state is meaningful to the model but is not necessarily directly interpretable by humans.

An explicit structure of objects and space – the model represents the world through identifiable entities and their relationships: objects, their properties, positions, geometry, and interactions. It carries this structured representation forward as the world changes.

But the lines between these three groups aren’t clear-cut: Dreamer imagines long sequences of actions, Waymo generates rare driving scenarios, and Atlas turns sparse observations into explorable space. Let’s look inside the architectures that make this possible →

Observation-generating world models predict what an agent could observe as the environment changes. You can see what they predict as images or videos, which makes them useful as learned simulators.

The basic process looks like past observations + conditioning → future observations. Conditioning may include factors including actions, camera trajectories, language instructions, etc.

There are two typical variants of architecture in this group:

This type of models typically work like this:

But on top of this workflow, developers add some smart architectural upgrades. Two very illustrative examples here are IRIS (Imagination with auto-Regression over an Inner Speech) and Dreamer 4 (by Google DeepMind). They are both reinforcement learning agents that learn game mechanics inside their own world models: IRIS for Atari and Dreamer for Minecraft. They first learn to predict how the game changes after each action, then let the agent practice inside these imagined game sequences. IRIS represents frames as discrete visual tokens and predicts them autoregressively with a GPT-like Transformer.

Dreamer 4 has some interesting twists in typical architecture. It combines a causal video tokenizer with an interactive dynamics Transformer. The tokenizer compresses frames into continuous latent states. When you move the mouse or press a key, the Transformer predicts what changes next. Dreamer takes just four refinement steps to produce the next state thanks to shortcut forcing objective. This make it real for Dreamer 4 to complete the 20,000-plus-action sequence needed to obtain diamonds without returning to the real game during training.

But the tokenizer is a serious bottleneck for this subtype of world models. If compression loses a small but critical object and dynamics, a model may never receive the information it needs to predict that object correctly.

These models generate observations through a series of refinements. Diffusion learns to reverse a noising process, and flow-based methods learn a transformation from a noise distribution toward the data distribution. Both can operate on compressed visual representations.

At prediction time, the model receives context and uses it to guide the generation of a possible future. Different noise samples can produce different outcomes from the same starting point.

Here, we want to look at systems that are already turning into something tangible.

Take DeepMind’s Genie 3 for example. It builds an interactive world by predicting the next frame from previously generated frames and the user’s latest action. This is called action-conditioned autoregressive generation: each prediction becomes part of the history used for the next one. That history helps preserve locations and objects when they leave and re-enter view, without requiring a predefined 3D scene.

And now take its real application – Waymo World Model, introduced in February 2026, adapts Genie 3 for autonomous-driving simulation. It generates camera and lidar observations conditioned on driving actions, scene layouts, and language instructions. Engineers can change the weather, introduce unusual obstacles, or test alternative routes to evaluate how the Waymo Driver handles situations rarely encountered on real roads.

Pretrained video diffusion models also serve as the backbone for several modern World Action Models (WAMs) which jointly generate robot actions and future video frames. We will talk about them in the next episode.

Now let’s look at flow matching variant. Cosmos-Predict2.5 is NVIDIA’s video world model for robotics, autonomous driving, and other Physical AI systems. Given text, an image, or a video, it generates a possible visual continuation of the scene, for example, how traffic may change as a car moves or how objects may react during a robotic task. A causal VAE first compresses the video into compact tokens, then a Diffusion Transformer uses flow matching to turn noisy tokens into a clean future sequence. Cosmos-Reason1 encodes the instruction and guides generation through cross-attention, while supplied image or video frames anchor the beginning of the sequence.

But the main limitation for this subtype is that generated futures may look realistic but be not physically right, especially as the generated sequence gets longer. Another cumulative error comes from wrong placement of the objects of the scene.

The second group of world models takes another idea as the foundation. Imagine that a robot needs to move a cup. First it needs to estimate what its movement will do. A latent dynamics model lets it explore possible outcomes through compact internal representations, without generating a detailed image of each one.

But besides efficiency from compression, the model needs accuracy from correct information that helps predict what happens next. The following three approaches use different training signals to make that happen.

Reconstructive models learn an internal state from which observations can be recovered. A decoder provides a straightforward check: does the compressed state contain enough information to reconstruct what the agent actually saw?

In PlaNet and DreamerV1–V3, the dynamics core is a Recurrent State-Space Model, or RSSM. Its state has two connected parts:

When the robot receives a new image, the model combines it with its memory to update its state estimate. When imagining the future, there is no new image available, and the model predicts the next state from its existing state and the proposed action.

Training brings these two routes together. The observation-informed state teaches the predictive route what it should have expected, while the decoder reconstructs the observation. A KL-divergence term encourages the two latent distributions to agree.

Such world models can also predicts rewards and whether an episode continues. In this case, actor usually proposes actions inside imagined rollouts, and its critic estimates future returns. The agent can therefore learn which actions work without decoding every predicted state into an image. Basically, reconstruction anchors the representation during training, while imagination runs through latent transitions.

JEPA (Joint-Embedding Predictive Architecture) is one of the most important ideas pushing predictive world models forward. Its premise is simple: instead of reconstructing every pixel, predict the abstract representation of what should be there.

Imagine that part of a video frame is hidden. A generative model may try to recreate the missing pixels, including textures, shadows, and other details that are difficult to predict and may not matter. JEPA works in representation space instead. A context encoder represents what the model can see, a target encoder represents the hidden target, and a predictor learns to bring the predicted and target representations close together. By focusing on features that remain stable and meaningful – such as objects, motion, and relationships – JEPA can learn compact representations useful for recognition, prediction, and, in action-conditioned variants, planning. The trade-off is that its predictions are harder for us to inspect because there is no decoded future frame to look at.

For a closer look at the architecture and the growing family of models, continue with our full guide on JEPA.

Task-oriented models, in turn, shape their internal states around evaluating actions. They learn to predict consequences like immediate rewards and longer-term returns.

The most illustrative example is MuZero that connects three functions.

It implement tree search approach to repeatedly apply these functions and explore alternative action sequences. Training checks its reward, value, and policy predictions against learning targets. The imagined state only needs to support those predictions, but doesn’t need to look like the environment.

Another variant, Newt, presented at ICLR 2026, develops this architecture with language conditioning, demonstration-based pretraining, and multitask reinforcement learning. The central mechanism remains a compact dynamics model used to evaluate possible actions.

This final group may seem to have the most advanced models. The thing is that structured world models make spatial organization or individual entities explicit parts of the representation. This introduces assumptions about the world: observations come from particular locations, objects persist, and interactions depend on relationships between entities.

What is most interesting – spatial and object-centric approaches are complementary. A spatial grid can represent a room without separating every object. An object-centric model can represent separate entities without assigning them explicit 3D geometry.

Spatial structure can take several forms. Depth maps describe distance from a camera. Occupancy grids divide space into cells and predict whether they are occupied, sometimes also identifying their semantic category. Point clouds, meshes, and Gaussian splats provide other ways to represent or render a scene.

Here we have the freshest example: World Labs’ Atlas, introduced in September 2026 by the company co-founded by Fei-Fei Li. They describes it as an autoregressive diffusion transformer with a shared spatial context that takes a really broader multimodal approach.

Its central idea is to give visual observations a place in space: an image comes with camera information that tells the model where it was captured and in which direction the camera was looking.

These observations form a shared spatial context. Imagine supplying photos of a room from opposite corners, for instance. Atlas can use their camera poses to relate the views, then generate what a camera placed between them would see.

Three architectural components work together:

There are two processes at the same time: autoregression determines how the sequence develops, while flow generates the content of each visual output. Camera geometry makes those predictions spatially grounded.

Atlas also handles depth maps, connecting image pixels to estimated 3D positions. Generated views and depth estimates support point-cloud reconstruction, which can be converted into Gaussian splats for rendering an explorable scene. With sparse inputs, unseen regions are inferred; additional observations constrain those guesses.

This combination places Atlas world model across two architectural groups: it generates observations, but organizes that generation around explicit spatial information.

This category is also very interesting. Object-centric models divide a scene into separate entity representations. Many use slots: learned vectors intended to capture individual objects. A dynamics model updates the slots over time and models their interactions.

SlotFormer is a foundational example. It starts from object representations extracted from video, then uses a transformer to predict their future evolution. The predicted slots can be decoded into frames or used for downstream reasoning and planning. Attention lets the model represent interactions between objects as well as changes within each object.

OC-STORM, with a revised paper in February 2026, combines object representations from a segmentation network with a learned world model. A small number of annotated frames helps identify decision-relevant objects and their interactions.

This structure is useful when a task depends on individual entities: which object moved, what it contacted, or where it should go next. The difficulty begins with identifying those entities reliably, especially when objects overlap, change appearance, or disappear behind one another.

To sum up what we have learned today:

But how to choose from all the spectrum of the groups and subgroups what will fit your workflow? Some practical advice:

The most interesting world models will not fit neatly into one box, as no today’s model does. World Labs’ Atlas, for example, combines generated observations with explicit spatial structure. Now the core direction is toward systems that mix these strengths: rendering enough to imagine, latent dynamics enough to plan efficiently, and structure enough to stay grounded in the physical world.

Next episode: Guide: VLAs vs. World Action Models: Should Robots React or Imagine?

A world model architecture defines how an AI system represents the current state of an environment and predicts how that state may change. Some models generate future observations directly, others predict compact latent representations, while structured models explicitly represent spatial geometry, objects, or relationships between entities.

A useful way to group world models is into three broad architectural families: observation-generating models, latent-dynamics models, and structured world models. These are not strict boundaries – many modern systems combine ideas from more than one group – but they capture the main difference in what each architecture carries forward through time.

Observation-generating models predict what the future will look like, such as the next image or video frame. Latent-dynamics models instead predict how a compressed internal representation of the world will change. This makes latent models more efficient for planning because they do not need to render every possible future in pixel space.

A video model becomes closer to a world model when it does more than generate visually plausible frames and starts modeling how an environment changes over time, particularly when predictions are conditioned on actions. Systems such as Genie 3 and Cosmos-Predict2.5 illustrate this direction, although visually realistic outputs are not necessarily physically correct.

It depends on the task. Latent-dynamics models are attractive for fast planning because they can simulate many possible futures cheaply. Structured models are useful when robots need explicit geometry or object relationships. Observation-generating models are valuable when visual simulation itself matters. In practice, advanced robotics systems may combine several of these approaches rather than choose only one.