World Models and World-Action Models: An Accessible and Comprehensive Survey

What should a world model predict?

The survey’s central question is: what should a world model predict, and should it also generate robot actions?

‍

Different systems answer the first part of this question in very different ways. Some compress observations into a small latent state and predict how that state evolves. Others predict only the rewards, values, and policies needed for planning. Transformer-based approaches turn images and experience into tokens and predict them like language, while diffusion models generate future frames through iterative denoising. Joint-embedding approaches such as JEPA avoid predicting pixels altogether. Instead, they forecast abstract representations of what matters in the scene. At the largest scale, video foundation models aim to become general-purpose simulators.

‍

Our survey paper groups these approaches into six categories: latent-recurrent state-space models, value-equivalent planning models, autoregressive-token transformers, diffusion and generative-frame models, joint-embedding predictive architectures, and video-generation foundation models. These categories aren’t tied to a single use case. The same basic approaches now appear across reinforcement learning, robotics, autonomous driving, games, and general-purpose simulation, as well as language, code, and agentic environments.

Predicting the future vs. carrying out actions

A world model predicts what happens next given the current situation, and it may propose an action. A world-action model also produces the actions needed to reach a desired future.

‍

That distinction is especially important in robotics, where real-world data is slow and expensive to collect. A robot can use a learned model to plan candidate motions, train a policy on imagined trajectories, or generate synthetic experience that expands a limited real-world dataset. Recent systems increasingly combine these functions, using one model as simulator, policy, and sometimes evaluator.

‍

Our paper argues that these architectural categories are also beginning to converge. Modern systems mix recurrent latent models with transformers, diffusion with autoregression, and world simulation with action generation. Rather than one architecture simply winning, the field is becoming a design space defined by what a model predicts, how it generates predictions, and whether it can be controlled through actions. The survey evaluates the state of the art by application domain and concludes that no single modeling approach dominates everywhere.

‍

Understanding physics

One of the most important questions in the survey is whether large video-generation models should be considered true world simulators. These systems can generate increasingly realistic and interactive environments, but visual quality alone does not guarantee that a model has learned the underlying laws of the world.

‍

Our paper reviews evaluations showing that video models can perform well on familiar situations while failing when physical conditions move outside their training distribution. A generated future may look convincing while violating conservation laws, object permanence, geometry, or causal relationships. This gap matters much more when a model is used to control a robot or vehicle than when it’s simply being used to generate video.

‍

This insight explains a deeper disagreement over prediction targets. Pixel-generative models bet that sufficiently large-scale prediction will result in physical understanding. Joint-embedding models instead predict abstract features, avoiding unpredictable visual detail. Value-equivalent approaches go even further, predicting only what is needed to make good decisions. The survey treats this not as a settled question, but as one of the field's defining research problems.

‍

Priorities for the next generation of world models

Our paper closes with a research agenda centered on the limitations we observe across approaches: long-horizon drift, hallucinated or physically inconsistent futures, weak memory and 3D persistence, imperfect controllability, the compute required for real-time interaction, incomplete evaluation, and safety as models gain more autonomy.

‍

Several priorities follow from those limitations. Future world models will need stronger physical and causal grounding, persistent representations of scenes over long time horizons, reliable conditioning on actions, and inference fast enough for interactive control. They will also need evaluation methods that measure more than visual realism, including physical consistency, controllability, planning usefulness, and performance under distribution shift.

‍

For robotics in particular, we see a lot of opportunity. A sufficiently accurate world model could let robots rehearse tasks safely in imagination, generate new training experience from limited field data, and evaluate possible actions before executing them. World-action models could bring prediction and control into a single learned system. But closing the gap between a plausible imagined future and a physically correct one will be essential for reliable real-world deployment.

‍

Download the full survey paper to explore the complete taxonomy and evaluation landscape of world models and world action models, and get in touch with our team.