∇ Guide: 3D Gaussian Splatting: How do robots remember what's no longer in view?

TL;DR: 3D Gaussian Splatting reconstructs scenes from multiple views using millions of small 3D primitives. Systems like GSMem turn these reconstructions into searchable spatial memory, allowing robots to revisit, update, and reinterpret previously observed places. Combined with additional physics models, Gaussian representations can also support simulation, but remembering a scene isn't the same as understanding how it behaves.

Since, we've dedicated this week to agentic memory, today we're jumping straight into one of its most interesting challenges: how can a robot remember a world it can no longer see?

When AI moves from a screen into the physical world, spatial intelligence becomes essential. A robot becomes a part of the environment – it is moving through it, changing its position, and interacting with objects.

But its “eyes”, its camera, only captures what is in front of it. What happens to everything else? How does a robot know where something is after turning away? And how can it return to a place it explored earlier? For that, it needs a map, a 3D map, of the surrounding space that it can keep updating and return to.

There are different ways to build such representations, but today we'll focus on a particularly interesting and hot one: 3D Gaussian Splatting (3DGS).

Originally developed to reconstruct photorealistic 3D scenes from images, Gaussian Splatting is now being explored as a foundation for spatial memory in embodied AI.

And a robot reconstructing a room is only the starting point. That same scene can be stored, searched, revisited from novel viewpoints, and tied directly into physical simulation.

Let's see how that works and how we can achieve Gaussian memory.

In today’s episode:

Let’s start with an everyday scenario from the near future. A robot enters a kitchen, sees a cup on the table, then turns toward the sink. The cup disappears from its camera view, but the robot still needs to know where it is.

A short history of camera frames can help to detect where the cup was. A robot’s camera sees the world one frame at a time. But the mug doesn’t stop existing when the robot turns away. To come back for it, the robot needs to remember where it was and how to get there.

A 3D memory connects separate observations into a space the robot can revisit. It preserves what things looked like, where they were, and how the surroundings fit together.

And what matters now may not have mattered a minute ago. A robot might pass a shelf without noticing the keys on it. If memory keeps only recognized objects, those keys may leave no searchable trace. Keeping the captured scene gives the robot another chance to find them when someone asks. It can examine that reconstruction from a different angle, within the limits of what it actually observed.

This memory also needs room for correction. Small errors in estimated movement accumulate. Recognizing a familiar place can help the robot realign its map and recover where it is.

A robot needs 3D memory because seeing something, understanding its relevance, and returning to it can happen at three different moments, and memory is what connects them.

3D Gaussian Splatting (3DGS), a breakthrough approach to 3D reconstruction, was introduced in August 2023 by researchers Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler, and George Drettakis from Inria, Université Côte d’Azur and Max-Planck-Institut für Informatik in the paper called “3D Gaussian Splatting for Real-Time Radiance Field Rendering”.

Let us define the terms first

Each Gaussian has a set of properties:

When the algorithm combines millions of these tiny clouds, layering them with different levels of opacity, they form a complete, remarkably realistic image.

Now let’s see how this actually happens →

Go deeper with the full guide. Follow how 3D Gaussian Splatting works, how robots search and update Gaussian maps, and what it takes to connect those maps to physical simulation. It’s fascinating!

Join Premium members from top companies like Microsoft, NVIDIA, Google, HF, OpenAI, a16z, plus AI labs such as Ai2, MIT, Berkeley, .gov, and thousands of others to really understand what’s going on in AI. 

This week we also recorded an overview of how Yann LeCun’s JEPA (H-JEPA, DeepJEPA, RoboJEPA, has evolved and what it can do →

3D Gaussian Splatting (3DGS) is a technique for reconstructing 3D scenes from images. It represents a scene using millions of small, overlapping 3D Gaussians, each with properties such as position, shape, opacity, and color. Together, they allow the scene to be rendered from new viewpoints in real time.

A Gaussian map becomes useful as spatial memory when a robot can continuously update it, retrieve relevant regions, and revisit previously observed places. Combined with semantic features and scene graphs, it allows robots to search their surroundings and examine details they may have missed during initial exploration.

Robots can use scene graphs to locate recognized objects and language-aligned features to search regions by meaning. Systems like GSMem combine both approaches, then render selected viewpoints for a vision-language model to inspect. This makes it possible to retrieve visual information even when the original object detection failed.

Yes, with additional mechanisms. Standard 3DGS reconstructs static scenes, but extensions can update Gaussian maps as observations change or connect Gaussians to physical particles. This helps keep the representation aligned with moving objects and, in some systems, supports simulation of their motion.

Not by itself. Gaussian Splatting primarily represents appearance and geometry. When combined with physical simulators and models that estimate material properties, it can help represent and predict how objects move, collide, or deform. However, reconstructing a scene doesn't automatically give a robot an understanding of its physical behavior.