∇ Guide: LeHami Explained – Can Physics Help World Models Predict Motion?

TL;DR: LeHami is a physics-inspired JEPA world model that uses Hamiltonian mechanics to prevent certain kinds of collapse during long-term latent predictions. It represents states using position-like and momentum-like variables, learns an energy-like function, and updates them through a symplectic leapfrog integrator. Experiments on simulated pendulums show promising gains in prediction accuracy and compute efficiency, although planning results are mixed and the approach hasn't yet been demonstrated on complex real-world dynamics.

What does 1830s physics have to do with Yann LeCun's JEPA world models? Quite a lot actually.

JEPA model architecture stands out for predicting abstract representations of what’s missing or what comes next, rather than generating images pixel by pixel. Though we’ve covered JEPA in many articles and videos before and explored its lore and the evolution roadmap, we’ve only scratched the surface of JEPA. There is a lot more beneath the surface for researchers to uncover and explore in JEPA.

Just recently, Yann LeCun, Romina Yalovetzky, and Randall Balestriero took JEPA optimization one step further. This time they turned to 1830s physics to create a new version of JEPA called LeHami.

“Le” in the name comes from LeJEPA version of JEPA, and “Hami” defines that it is based in Hamilton physics. The idea is to keep long predictions from gradually losing distinctions between different states. That is why LeHami changes how predictions are made, building a geometric constraint borrowed from Hamiltonian mechanics directly into the process.

But why exactly Hamiltonian mechanics? How do these physical constraints shape the JEPA architecture, and what does that buy us in the end?

These are the questions we’ll answer today, brushing up on some classical mechanics along the way. Let’s dive into how LeHami brings the two together.

In today’s episode:

If we look at JEPA’s prediction process, we find a risky, creeping problem at its foundation. During a long rollout, each predicted representation becomes the input for the next prediction. If the predictor slightly compresses the range of possible states at every step, that compression tends to compound. As a result, distinctly different starting states end up represented more and more similarly. Even if the encoder learned useful representations during training, the predictor still runs the risk of collapsing over time.

With LeHami, the researchers aim to rethink how those representations evolve over time, borrowing structure from Hamiltonian mechanics – a framework for describing motion developed almost 200 years ago.

William Rowan Hamilton introduced its vision of mechanics in two major papers published in 1834 and 1835, extending mathematical methods he had used in optics to mechanics. The goal was a more general, systematic way to calculate motion, including complicated systems like interacting planets. Some people may think that it was a new set of physical laws, but actually it was a reformulation of existing mechanics.

So what changed?

Classic Lagrangian mechanics describes a system through its coordinates and velocities. For familiar mechanical systems, its central function, the Lagrangian, is kinetic energy minus potential energy. This function gives us equations describing how the system accelerates.

Hamiltonian mechanics switches from velocities to momenta. Coordinates q and momenta p together describe a point in phase space. Its central function, the Hamiltonian H, is often the total energy: kinetic plus potential. The gradients of H tell us how q and p change over time.

The best illustration of it is how a pendulum works:

But why do we need this specific change?

For simplicity of equations. Instead of second-order equations for coordinates, we get paired first-order equations for coordinates and momenta. Both approaches can describe the same motion, but Hamilton’s version makes the geometry of its evolution more explicit.

Standard JEPA doesn’t require Hamiltonian mechanics. But LeHami’s researchers wanted a predictor with a specific property: it shouldn’t gradually squeeze a whole region of latent states into a smaller volume as predictions feed into one another.

The Hamiltonian formulation gives them a direct way to build that constraint into the update:

That is why the authors choose Hamiltonian rather than Lagrangian mechanics here. It offers a computationally simple route to the exact guarantee they need, rather than a debate over which formulation is more valid.

Kinetic energy relates to motion, while potential energy relates to position. LeHami borrows that structure, learning $K$ and $V$ in latent space instead of computing actual physical energies.

Now let's see how LeHami puts these physics-inspired updates into reality →

Go deeper with the full guide. See how LeHami turns Hamiltonian mechanics into a JEPA predictor, why its symplectic updates prevent certain kinds of latent collapse, and what the experiments reveal about its strengths and limitations.

This week: 20% off Premium. Offer ends October 25 at 11:59 pm ET

Join Premium members from top companies like Microsoft, NVIDIA, Google, HF, OpenAI, a16z, plus AI labs such as Ai2, MIT, Berkeley, .gov, and thousands of others to really understand what’s going on in AI. 

Not interested in the topic, check our latest about the new world that DeepSeek is building →

LeHami is a JEPA-based world model that uses Hamiltonian mechanics to structure how its latent states evolve over time. Instead of relying entirely on an unconstrained neural predictor, it learns a Hamiltonian function and uses a symplectic update rule to preserve the geometry of its latent state space.

JEPA is a general framework for predicting representations rather than reconstructing every pixel. LeHami introduces a particular physics-inspired structure for sequential latent prediction. It uses position-like and momentum-like variables and updates them according to learned Hamiltonian dynamics, reducing the risk of certain kinds of prediction collapse.

Hamiltonian mechanics provides a way to describe dynamics using paired position and momentum variables. Its symplectic structure preserves phase-space geometry and volume, which helps prevent different latent states from collapsing together during repeated prediction steps.

A symplectic integrator is a numerical method designed to preserve the geometric structure of Hamiltonian dynamics. LeHami uses a leapfrog integrator that updates its momentum-like and position-like variables in three stages. This makes it possible to advance the learned state while preserving the relevant symplectic structure.

Not necessarily. LeHami learns energy-like functions in latent space, but these aren't guaranteed to correspond to the actual physical energy of a scene. The architecture borrows mathematical structure from physics without requiring explicit labels for real-world positions, velocities, or energies.

The initial experiments show improvements in some prediction tasks and compute efficiency, but the results are mixed. LeHami performed well on simulated pendulum prediction, while a stronger comparison model achieved better results on some planning tasks. Its effectiveness on more complex environments remains an open question.