Artificial Intelligence1 min read

After 600 experiments, here's what I discovered about JEPA world models

By · Published by Everything Blog

In short

The LeWM model functions entirely within an abstract embedding space using an encoder, predictor, and planner rather than generating full pixel predictions. While the vanilla architecture dropped from an 89.7% success rate on 25-step tasks to just 22% on unguided 75-step tasks, quadrupling search iterations failed to resolve the issue. By introducing hierarchical subgoals alongside an automated latent memory graph built from roughly 460,000 demonstration states, researchers boosted unguided long-horizon performance to 85%.

Key points

  • Scaling brute-force search compute fails to fix execution errors in long-horizon JEPA pla…: Scaling brute-force search compute fails to fix execution errors in long-horizon JEPA planning tasks.
  • Hierarchical segmentation allows abstract world models to maintain high accuracy over ext…: Hierarchical segmentation allows abstract world models to maintain high accuracy over extended operational sequences.
  • Automated latent memory graphs eliminate the need for live expert demonstrations while su…: Automated latent memory graphs eliminate the need for live expert demonstrations while substantially boosting autonomous performance.

Sources checked by our analysis pipeline: 1

World models · JEPA · Planning

After 600 experiments, here's what I discovered about JEPA world models

There's a big debate in AI right now about what a world model should be. Most of the attention goes to generative world models, which predict the future by generating it, usually in pixels. Many scientists, including Yann LeCun, argue that this is wasted effort. Most of the pixels don't matter for deciding what to do, and many of them can't be predicted at all. [1]

The proposed alternative is JEPA, the joint-embedding predictive architecture. It has three parts: an encoder, a predictor and a planner. The encoder turns each frame of observation into a learned embedding, a compact vector that describes the scene. The predictor takes that embedding plus an action and predicts the embedding of the next frame. The planner then tries hundreds of action sequences inside the predictor and picks the one whose predicted end state lands closest to the embedding of the goal. The whole loop runs in embedding space, so the model only has to learn the parts…

Artificial Intelligence