World Models Interview Cheat Sheet

§0 TL;DR Cheat Sheet

1. Three lineages: (1) latent dynamics / model-based RL — learn a compact state space you can roll out inside; (2) predict-in-representation-space (JEPA) — predict only the encoder's output, never reconstruct pixels; (3) generative video and interactive simulators — generate the interactive future observation itself.
2. Lineages are not camps: Dreamer 4 is both a generative dynamics model and imagination RL; Cosmos 3 spans understanding, generation and action; video diffusion models also work inside a VAE latent — "latent" is not a word owned by RSSM or JEPA.
3. State: the environment's true state $x_t$ is unobservable; the model state $s_t=(h_t,z_t)$ is its stand-in. The RSSM recursion is written $h_t=f_\theta(h_{t-1},z_{t-1},a_{t-1})$ throughout: $h_t$ supplies memory across time, $z_t$ carries stochasticity and new information.
4. prior vs posterior: $p_\theta(z_t\mid h_t)$ is used for prediction, $q_\phi(z_t\mid h_t,o_t)$ absorbs the current observation. Only the prior may be used inside imagination — calling the posterior means peeking at an observation that has not happened yet.
5. KL balancing: $\mathcal L_{KL}=\beta_{dyn}D_{KL}(\mathrm{sg}\,q\Vert p)+\beta_{rep}D_{KL}(q\Vert\mathrm{sg}\,p)$. Both terms are in the $q\Vert p$ direction; what is split is whose gradient is stopped, not the KL direction, and certainly not "turn the coefficient down".
6. Four decision interfaces: PlaNet plans online in latent space; Dreamer trains an actor-critic inside imagination; MuZero uses MCTS and reconstructs no observation at all; TD-MPC2 learns dynamics, reward and value decoder-free for MPC to use.
7. The shared JEPA structure: $P_\theta(E_\phi(x),m)\to\mathrm{sg}(E_{\bar\phi}(y))$ — an asymmetric predictor + an EMA target branch + stop-gradient. This combination is not a no-collapse theorem: a constant encoder with a constant predictor is still a degenerate solution.
8. What matters on the generative route is not image quality but the action interface: a latent action model's small codebook, camera pose, text conditioning and actuator commands mean completely different things; action-free video supplies only a prior — to use it for action selection you still have to add the interface and the evidence that goes with it.
9. Numbers come with provenance: Genie 3's 720p / 24 FPS / few-minutes consistency all come from the official blog post of 2025-08-05 — no paper, no published parameter count (11B is Genie 1's); GameNGen's speed is reported per version — the v1 abstract says ">20 FPS", the v2 body gives 20 FPS on a single TPU-v5 with 4-step DDIM — and 29.4 dB is the PSNR of next-frame prediction, not the fidelity of a long rollout, and still less evidence of physical correctness.
10. Killer questions: why KL balancing is not the same as shrinking the coefficient, what exactly is zero in V-JEPA 2-AC's zero-shot, and whether photorealistic video can prove an understanding of physics — Physics-IQ gives an experimental answer (across models $r=-0.46$, $p=0.249$, no significant correlation detected), not a verdict that the two are unrelated.

§1 One question, three lineages

1.1　What a world model is

A world model is an agent's internally learned model of the environment's structure and dynamics: given the observations and actions so far, it can predict "if I do this, what happens next". The definition deliberately leaves the output form open, because which output is necessary depends on the use — to do RL inside the model you need a way to compute returns in imagination and to truncate episodes (the Dreamer family does this by learning a reward head and a continuation head; when the reward function or the termination rule is known, using the known one is equally fine); for visual goal-directed navigation you only need to compare distances between goal representations; to serve as a playable simulator you must generate observations a human can look at.

So "does it count as a world model" is mostly a fake question. The real ones are three: what information is kept, which quantity is predicted, and which kind of decision that quantity can support.

1.2　Three lineages (they overlap)

1.3　Selection criterion

"Priority goes to work that establishes a key mechanism, represents a major technical route, or was released by a major research group and is closely tied to embodied decision-making; at equal representativeness, open weights come first. The newest releases are used to indicate direction, and their maturity is stated separately."

The work that was excluded and the reason for each entry are in §A.2 — when an interview asks about a name that was not included, knowing why it was excluded is more useful than memorising one more name.

1.4　Term disambiguation (where interviews most often talk past each other)

• The environment model of RL: $p(s_{t+1}, r_{t+1} \mid s_t, a_t)$, in service of planning and imagination. Lineage one.
• The world model in LeCun's sense: prediction in an abstract representation space, explicitly refusing pixel-level reconstruction. His position paper was published on OpenReview (v0.9.2, 2022-06-27) and has no arXiv number — don't invent one when citing it. Lineage two.
• "World simulator": a generative model that can produce interactive video conditioned on actions. Lineage three.
• The "world model" inside an LLM: whether a language model's activations encode external state such as a chess board or a map. That asks "is it in the representation", not "can it predict the consequence of an action". A different question, outside the scope.
• "World model" in the sense of code execution: a model that predicts program execution state. This is a terminology collision with the world models discussed here, not the same line of work.

Two more pairs of symbols and scopes that must be kept apart:

• $x_t$ vs $s_t$: $x_t$ is the environment's true state, unobservable, and the model makes no promise to recover it; $s_t$ is the model's own state. Saying "the latent state is the true state" is wrong.
• Static representation vs interactive dynamics: I-JEPA is a static image representation method and does not by itself constitute an interactive dynamics model; calling it "the image version of a world model" collapses under the first follow-up.

§2 Convention (used throughout)

2.1　Time convention

One timeline throughout: receive $o_t, r_t, c_t$ → form $s_t$ → execute $a_t$ → receive $o_{t+1}, r_{t+1}, c_{t+1}$.

So $r_{t+1}$ is the immediate reward of action $a_t$, and $c_{t+1}\in\{0,1\}$ flags "the episode has not ended at $t+1$". Index shifts of one between papers almost always come from a different version of this convention — align the convention first, then compare formula coefficients.

2.2　Symbol table

2.3　Prediction heads: which ones are necessary

A Dreamer-style world model usually carries three heads: observation (reconstruction or token prediction), reward and continuation. The latter two exist so that "RL inside the model" can run — in this Dreamer implementation, returns in imagination come from the learned reward head and episode truncation from the learned continuation head; remove either one and the computation no longer closes.

That necessity holds only for this implementation; don't promote it into a general law. When the reward function or the termination rule is known (which is true of many simulators and games), calling the known one computes returns and truncates episodes in imagination just as well. Observation reconstruction is even further from necessary for "RL inside the model" — MuZero and TD-MPC2 both plan without reconstructing observations (§3.8).

A pure video generation model does not necessarily need reward and continuation: its purpose is to generate a future you can look at. Treating "no reward head" as a defect, or "has a reward head" as the definition of a world model, both give wrong answers.

2.4　Number provenance

Every external number states, at its first appearance in the body, the task, metric, model version and setting; speed numbers name the hardware; improvements distinguish percentage points from relative percentages. Openness is reported per artefact — paper, code, weights and data are four different things, and a planned release is not written as already open. Claims supported only by an official blog post or press release are always labelled "blog-only".

§3 Lineage one: latent dynamics and model-based RL

3.1　The starting point: training a policy in a dream

Ha & Schmidhuber 2018 (World Models, arXiv 1803.10122) split the structure into three pieces: a VAE compresses each frame into a low-dimensional $z$; an MDN-RNN predicts the next-step mixture-of-Gaussians distribution over the $z$ sequence; and a linear controller reads an action out of $(z,h)$. The key step is the last one — in the paper's VizDoom experiment the controller is trained entirely inside rollouts generated by the RNN (the paper calls them "dream"), then put back into the real environment; the CarRacing experiment is not like this, its controller is trained directly in the real environment. "The policy is trained entirely in a dream" is true only of the former.

Those three pieces set the division of labour that followed: representation (compress the observation into something predictable), dynamics (extrapolate on that quantity), and decision (read an action out of the state). Later work either changes one of them or merges them.

3.2　RSSM: why the state is half deterministic, half stochastic

PlaNet (Hafner et al., arXiv 1811.04551) introduced the Recurrent State-Space Model:

$$h_t=f_\theta(h_{t-1},z_{t-1},a_{t-1}),\qquad z_t\sim p_\theta(z_t\mid h_t)\ \text{(prior)},\qquad z_t\sim q_\phi(z_t\mid h_t,o_t)\ \text{(posterior)}$$

$h_t$ is deterministic: it provides a recurrent memory channel that does not pass through random resampling. A purely stochastic state has to be resampled at every step, and over a long horizon the information is washed out by sampling noise. Don't read "deterministic" as "lossless" — a fixed-width GRU state still compresses and forgets; it is merely no longer scoured by per-step sampling noise. Nor does this say other architectures cannot have memory — a transformer looks back at history directly through attention, so memory exists there too, it is just not squeezed into one recurrent vector.

$z_t$ is stochastic: the environment itself is stochastic and observations are incomplete, so $z_t$ carries "the genuinely new information at this step". Don't take the opposite extreme either — deterministic dynamics can work very well in a deterministic environment; the value of $z_t$ rises with environment stochasticity and with the degree of partial observability.

The division of labour between prior and posterior is the core of the whole lineage: the posterior sees the observation, the prior does not. Both exist during training; at rollout time there is no future observation, so only the prior is left.

3.3　The objective: start from the negative ELBO with no coefficients

$$\mathcal L_t=\underbrace{-\log p_\theta(o_t\mid h_t,z_t)}_{\text{reconstruction NLL}}+\underbrace{D_{KL}\big(q_\phi(z_t\mid h_t,o_t)\,\Vert\,p_\theta(z_t\mid h_t)\big)}_{\text{consistency term}},\qquad z_t\sim q_\phi(z_t\mid h_t,o_t)$$

This is the single-step, single-sample estimate of the standard sequential-VAE negative ELBO: the reconstruction term should be $\mathbb E_{z_t\sim q_\phi}[\cdot]$, and one $z_t$ sample stands in for the expectation; the full objective also sums over $t$ along the whole sequence. Written out it is $\sum_t \mathbb E_{q}\left[\mathcal L_t\right]$, with no weights whatsoever. It reads as two sentences: the decoder must be able to restore the observation from $(h_t,z_t)$; and the prior must be able to guess what the posterior guessed, without seeing the observation.

Every KL weight, KL balancing and free-bits trick that follows is an engineering change to the relative influence of two objectives during optimisation — not a new probabilistic derivation.

3.4　KL balancing and free bits

DreamerV2 (Hafner et al., arXiv 2010.02193) splits the KL in two:

$$\mathcal L_{KL}=\beta_{dyn}D_{KL}(\mathrm{sg}\,q\Vert p)+\beta_{rep}D_{KL}(q\Vert\mathrm{sg}\,p)$$

Both terms are in the $q\Vert p$ direction. Balancing does not touch the KL direction; it decides where the gradient goes: the first term stops $q$'s gradient so only the prior chases the posterior (dynamics learning); the second stops $p$'s gradient so only the posterior concedes to the prior (representation regularisation). Taking $\beta_{dyn}\gg\beta_{rep}$ says: let the dynamics learn to predict first, then let the representation give ground for the sake of predictability.

sg targets the computation path. $q$ and $p$ share $h_t$, and $h_t$'s gradient still flows back along the other path — in code this is a new distribution built with probs.detach(), not some block of the network set to requires_grad_(False).

free bits is a different thing: $\max(\tau, D_{KL})$ flattens the KL at a floor $\tau$, so once the KL is small enough it is not pushed further and the spare capacity goes to reconstruction. It applies to the joint KL — sum over all latent groups first, then truncate; it is not a truncation per categorical probability.

DreamerV3 paper v1 (Hafner et al. 2023, arXiv 2301.04104, Eq. 4 and Table W.1) takes $\beta_{dyn}=0.5$, $\beta_{rep}=0.1$; the v2 body of the same paper writes $\beta_{dyn}=1$, $\beta_{rep}=0.1$. Coefficients are taken from v1 — when quoting coefficients, always say which version.

3.5　Where actions come from: online planning vs an actor-critic inside imagination

PlaNet trains no policy network. At every step it plans online in latent space using CEM: sample a batch of action sequences from a Gaussian, roll them forward with the prior, score them with the reward head, take the elite sequences with the highest returns and refit the distribution to them, and iterate for several rounds; finally execute the first action of the final round's distribution mean, then start over at the next step. Note it is not "sample once and execute the best sample" — iterative refitting is the substance of CEM. Planning never decodes an image, so the sample count can be set very large.

PlaNet also has latent overshooting: instead of only pulling the one-step prior close to the posterior, it pulls multi-step unrolled priors close to the posteriors at the corresponding times. It is a multi-step consistency signal, not "make the training horizon longer" — a model that is accurate one step at a time can still drift into absurdity after a few dozen compositions, and overshooting penalises that drift directly.

Dreamer (Hafner et al., arXiv 1912.01603) changed the decision interface: train an actor-critic inside imagination. Take a real segment from the replay buffer, use the posterior to get a starting point $s_t$, then unroll $H$ steps forward with the prior only, and update policy and value on those imagined states. Policy updates no longer consume real interaction, so sample efficiency rises sharply; the price is that the policy has only ever seen what the model believes will happen.

The λ-return, written under the time convention of §2.1:

$$G_t^\lambda=\hat r_{t+1}+\gamma\hat c_{t+1}\big[(1-\lambda)v(s_{t+1})+\lambda G_{t+1}^\lambda\big],\qquad G_H^\lambda=v(s_H)$$

The critic's regression target takes a stop-gradient. DreamerV3 v1 Table W.1 takes $H=15$ while the body's prediction sequence length is $T=16$: action transitions are always counted by $H$, so the state sequence including the start point has $H+1$ states. That difference of 1 is not a typo, it is "counting states versus counting transitions".

3.6　Discrete latents and DreamerV3's three pieces of engineering

DreamerV2 changed $z_t$ from a Gaussian to a set of categorical distributions (several groups, each a one-hot over several classes), sampled in the forward pass and approximated with straight-through in the backward pass. That is a biased gradient estimate — it passes gradients through a non-differentiable sampling operation as if it were the identity.

Mixing 1% of a uniform distribution into the categorical probabilities is a DreamerV3 change (v1 body and Appendix C), not a DreamerV2 one — don't cross the versions when quoting; its purpose is to stop some class probability being squeezed to 0 and blowing up the $\log$. Uniform mixing and free bits are two independent mechanisms: one prevents probability degeneration, the other prevents the KL from being pressed too hard.

Besides that uniform mixing, DreamerV3 adds three more mutually independent pieces of engineering:

1. symlog: $\mathrm{symlog}(x)=\mathrm{sign}(x)\log(1+\lvert x\rvert)$, inverted by symexp. It compresses signals spanning orders of magnitude into a trainable range, symmetrically and approximately as the identity near 0.
2. twohot: encode a continuous scalar onto a fixed set of bins — find which two neighbouring bins it falls between and split a weight of 1 between them by distance. Regression thus becomes classification, and the expected value is recovered exactly.
3. return normalization: take the difference between the 95th and the 5th percentile $S$ of a batch of returns, smooth it with an EMA, and divide by $\max(1,S)$. The floor of 1 is the crux: in sparse-reward tasks almost all returns are 0, so $S$ is tiny, and without the floor the noise of near-zero returns would be amplified into enormous gradients. This is not ordinary advantage normalisation (subtract the mean, divide by the standard deviation) — it only rescales, it has a floor, and it is smoothed across batches.

In v1 (the body after Eq. 10 and Appendix C), both reward and critic use symlog twohot: symlog compresses first, then twohot encodes into a categorical regression. Do not generalise this to every prediction head — the observation and continuation heads are another matter.

The most frequently misquoted thing about DreamerV3 is its headline sentence: one set of hyperparameters covering more than 150 tasks. That says there is no need to tune per environment; it does not say one set of weights handles every environment (each task is still trained separately). Collecting a diamond from scratch in Minecraft was achieved through interaction with the environment, not from offline data.

3.7　Dreamer 4: after the backbone is swapped

Dreamer 4 (arXiv 2509.24527) is no longer an RSSM: tokenizer + transformer dynamics + shortcut forcing. Don't drape V3's RSSM/KL/categorical-latent description over it; that is a different backbone.

Its most eye-catching conclusion must be memorised together with its qualifiers: the diamond is obtained under the paper's offline Minecraft setting, and the training pipeline includes behaviour cloning, reward modelling and RL inside the model; the "real time on a single GPU" the paper reports refers to interactive inference, not training.

3.8　Planning without reconstruction: MuZero and TD-MPC2

MuZero (Schrittwieser et al., arXiv 1911.08265) learns only the quantities planning needs: reward, value and policy prior, with MCTS search. It reconstructs no observation at all and does not require the latent state to recover $x_t$. The core is predictive sufficiency for planning — as long as those quantities are predicted accurately inside the search tree, the search can pick the right action. This is a design principle plus an empirical result, not a strict value-equivalence theorem.

TD-MPC2 (Hansen et al., arXiv 2310.16828) carries the same idea into continuous control: learn latent dynamics, reward and value decoder-free and use them for MPC, with no observation reconstruction. In the paper's scaling experiments, a single 317M-parameter agent covers 80 tasks.

3.9　What to do about model error: PETS and MBPO

These two are often mentioned together, yet they do not solve the same problem:

• PETS (Chua et al. 2018, arXiv 1805.12114) handles uncertainty representation: an ensemble of probabilistic models expresses both environment noise and the uncertainty caused by insufficient data, and trajectory sampling propagates it during planning.
• MBPO (Janner et al. 2019, arXiv 1906.08253) handles error accumulation: instead of rolling out from scratch, it branches very short model rollouts from states in the real data and then does off-policy updates, keeping accumulated model bias within an acceptable range.

In one line: PETS says "I'm not sure", MBPO says "I only dare look a few steps ahead".

3.10　What imagination generates: tokens or pixels

IRIS (Micheli et al., arXiv 2209.00588) imagines over token sequences with a discrete tokenizer + transformer; DIAMOND (Alonso et al. 2024, arXiv 2405.12399) uses a diffusion world model to generate image-level imagined trajectories, and its experiments support one concrete conclusion: visual detail affects control returns — the small objects and fine edges that get compressed away are often exactly what triggers a reward or a collision.

DayDreamer (Wu et al., arXiv 2206.14176) is this lineage's representative on real hardware: in that paper's experiments, a quadruped learned to walk with about one hour of real interaction. That number belongs to that experiment; it cannot be read as "model-based RL learns any skill in an hour".

§4 Lineage two: predicting in representation space (JEPA)

4.1　The shared structure

LeCun's position paper (OpenReview v0.9.2, 2022-06-27) argues that prediction should not happen on pixels, because most of the pixel detail of the future is neither predictable nor worth predicting. The JEPA family turns that argument into a trainable structure:

$$P_\theta\big(E_\phi(x),\,m\big)\ \longrightarrow\ \mathrm{sg}\big[E_{\bar\phi}(y)\big]$$

$x$ is the context, $y$ the target, $m$ the condition describing "which part to predict" (mask tokens, positions, etc.), $E_{\bar\phi}$ is an EMA copy of the online encoder, and the target branch takes a stop-gradient. Papers differ in three places — how the distance is computed, how the mask is built, how the target is constructed — discussed separately below; don't generalise any one paper's details into "that's how all JEPAs work".

4.2　I-JEPA: representation prediction on static images

I-JEPA (Assran et al. 2023, arXiv 2301.08243) works on a single image: take several target blocks from the image plus a context block, and use an asymmetric predictor to predict each target block's representation from the context representation. The loss is the squared $L_2$ between patch representations; the target branch comes from the EMA encoder and is not back-propagated; masking is a multi-block strategy (target blocks large enough, context block not overlapping the targets).

4.3　V-JEPA: moving to video

V-JEPA (Bardes et al. 2024, arXiv 2404.08471) moves the same structure to video: masked feature prediction, with the distance switched to $L_1$, still with stop-gradient + EMA target + predictor. It needs no negative samples, does no pixel reconstruction, and requires no pre-trained image encoder — those three "no"s are its position relative to contrastive learning and reconstruction-based self-supervision.

Note the prediction here is still action-free: the model learns to predict masked spatio-temporal representations (mask prediction, where masked blocks may lie before, inside or after the context in time), not to "predict the future causally given the past"; and there is certainly no "what I did" in it.

4.4　What actually prevents collapse

The standard answer is a set of mechanisms: the asymmetric structure (a predictor on one side only), the slow EMA target branch, stop-gradient, and the difficulty of the masking task itself. But the point is the next sentence:

4.5　V-JEPA 2 and 2-AC: action enters in the second stage

V-JEPA 2 (Assran et al. 2025, arXiv 2506.09985) is still pre-trained with action-free masked prediction ($L_1$, EMA target, stop-gradient). The 77.3 it is so often cited for is the top-1 accuracy of an attentive probe on Something-Something v2 — a representation-quality metric, not a robot success rate.

Action enters in the second stage: V-JEPA 2-AC freezes the pre-trained encoder and trains only an action-conditioned predictor whose input includes robot actions and end-effector state. The paper's "<62 h unlabeled robot video" means no task-semantic labels, not no action records — an action-conditioned predictor cannot be trained without actions.

Its planning interface is MPC in representation space: given a goal image, encode it into a goal representation; predict the representation reached after executing each candidate action sequence and minimise the $L_1$ distance to the goal representation; search action sequences with CEM, execute the first action and replan. The goal representation is not a reward label — it is one end of a distance being minimised, with no return regression anywhere.

4.6　V-JEPA 2.1: what changed, and how to break down the gains

V-JEPA 2.1 (arXiv 2603.14482) has three increments: apply prediction supervision to both masked tokens and visible-context tokens (the context term carries its own weight); use multi-layer outputs for deep self-supervision; and extend the image/video tokenizer and the model scale (the smaller sizes involve distillation).

The robotics gains must be reported broken apart: in the paper's Table 6, under the same planning setting, grasp goes from 60% to 70% (+10 percentage points); after changing the planning setting and lengthening the horizon it reaches 80% (+20 percentage points, at which point it is no longer a pure representation comparison). 10 tasks per skill. Blending the two tiers into "a 20% improvement" is the classic provenance accident — it mixes percentage points with relative percentages and mixes two settings at once.

4.7　DINO-WM and one common misattribution

DINO-WM (Zhou et al. 2024, arXiv 2411.04983) learns dynamics on frozen DINOv2 features, and plans towards the features of a goal image. What it demonstrates is: the representation can be off the shelf, and the dynamics is the part you have to learn — there is no need to retrain an encoder just to build a world model.

Navigation World Models (Bar et al. 2024, arXiv 2412.03572) is often filed under JEPA, but it does not belong here: it is a conditional diffusion transformer over first-person video and actions, and what it predicts is the future observation, which puts it in lineage three (see §5.2). Both do "goal-conditioned navigation planning", but they predict different quantities — the most decisive of the three questions in §1.1.

4.8　Theory progress and its boundary

A 2026 paper on JEPA generalisation (arXiv 2606.27014) connects pre-training error to the regret of downstream planning, in a setting of low-rank factorisation of a conditional spectral graph / action-conditioned co-occurrence matrix. The conclusion holds only under those assumptions — it is not an unconditional guarantee that "small representation loss implies good planning". Cite it together with its setting, or you are substituting a paper's title for an argument.

VL-JEPA (arXiv 2512.10942) brings language into the same framework; only the direction is mentioned here, with none of its numbers cited.

4.9　What this lineage is missing

Read against lineage one, the gap is clear — confined to the planning instance above (2-AC's representation-space MPC): no reward, no continuation, no explicit uncertainty, with the decision resting entirely on one quantity, the distance to the goal representation. That is a property of that interface, not a law of the whole lineage: reward or uncertainty estimation can perfectly well be attached on top of the same representation.

It is therefore naturally suited to goal-conditioned tasks (move the gripper to this visual goal), and when long-horizon returns or risk aversion are needed you have to supply the missing quantities yourself. Don't count "sparse reward" among the exclusions either — sparse-reward tasks can often be written in goal-conditioned form, which makes goal-conditioned MPC the convenient choice.

It also inherits an evaluation gap: probe accuracy measures whether the information is in the representation, not whether the consequences of actions are predicted correctly. So papers on this line must give both kinds of evidence — representation metrics and closed-loop success rates; giving only the former shows only that the representation is good.

§5 Lineage three: generative and interactive world models

5.1　The action interface: the real technical core of this lineage

Video generation models carry no action by nature. How to plug "what I did" into them is the most valuable stretch of this lineage.

latent action model (LAM): infer an "action" backwards from adjacent observations. The example here is the Genie-style discrete VQ LAM (DreamGen uses the continuous pre-quantisation embedding as a pseudo-action, see §6). The VQ bottleneck does three things:

$$k=\arg\min_j\big\lVert E(o_t,o_{t+1})-e_j\big\rVert^2$$

The encoder sees $(o_t,o_{t+1})$ and emits a continuous vector, which is quantised to the nearest code $e_k$ in the codebook; the decoder predicts the future observation from the current information and $e_k$; the training loss has three terms — a prediction term, a codebook term and a commitment term (straight-through is the standard VQ formulation, code in §9.3).

Why must the codebook be small? A capacity argument: one code carries at most $\log_2 K$ bits, which is 3 bits at $K=8$. Narrow the bottleneck until it only fits "the few directions that best explain the change between frames", and the model has no room left to stuff a whole frame's appearance difference through it.

Genie (Bruce et al. 2024, arXiv 2402.15391) is the first paper to scale a LAM into a product: 11B parameters, three components — a video tokenizer, a latent action model with 8 codes, and a dynamics model; generation uses MaskGIT-style parallel masked sampling, not GPT-style token-by-token decoding. Keep those three components separate; fused into "Genie is a video model", you cannot explain why it is interactive.

Besides LAM there are three other interfaces, semantically completely different and not interchangeable:

5.2　Real-time interaction and memory

GameNGen (Valevski et al. 2024, arXiv 2408.14837) shows a diffusion model can serve as an interactive engine, and both the speed and the fidelity number must be memorised together with their scope: the speed is written "over 20 FPS" in the v1 abstract, while the v2 body gives 20 FPS on a single TPU-v5 with 4-step DDIM sampling; the 29.4 dB PSNR is a next-frame prediction metric measuring single-step prediction, not the fidelity of a long rollout. The point is that these two numbers measure the speed and fidelity of simulating the game DOOM, not physical correctness — DOOM's "physics" is itself a set of game rules.

Genie 3 (Google DeepMind official blog, 2025-08-05, blog-only): 720p, 24 FPS, with consistency officially described as on the order of "a few minutes" and visual memory of about one minute. There is no paper and no published parameter count — 11B belongs to Genie 1, don't graft it onto Genie 3, and don't use Genie 1's three-component structure to infer Genie 3's internals either.

Project Genie (official blog, 2026-01-29, blog-only) is a consumer-facing prototype; the 60-second world length it announces is a limit of the product prototype, not a research ceiling for Genie 3.

Matrix-Game 3.0 (arXiv 2604.08995) puts the weight on three pieces of engineering: action control, memory retrieval, and streaming generation. The speed scope must be complete: a 5B model reaching at most 40 FPS @ 720p under asynchronous deployment, where that deployment uses 8 GPUs running the DiT + 1 doing VAE decoding. Writing it as "40 FPS on one GPU" is wrong.

Navigation World Models (arXiv 2412.03572) trains a conditional diffusion transformer on first-person video and actions, and uses it to evaluate candidate trajectories for goal-directed navigation — an example of a generative model acting directly as a navigation planner.

5.3　Multimodality and spatial conditioning

UniSim (Yang et al., arXiv 2310.06114) proposes the "universal action-conditioned simulator" route: unify data from different sources (human activity video, robot data, navigation data and so on) into an "observation–action–observation" conditional generation problem, and train an interactive video simulator.

Cosmos is a family, and cannot be compressed into one architecture:

• Cosmos WFM (NVIDIA 2025, arXiv 2501.03575) gives both an autoregressive and a diffusion path, and treats Predict / Transfer / Reason as different tasks rather than three modes of one model.
• Cosmos-Reason1 (arXiv 2503.15558) is a vision-language model for physical common sense and embodied reasoning; it cannot be used as evidence of a "dynamics model" — it answers questions, it does not predict the consequences of actions.
• Cosmos-Predict2.5 (arXiv 2511.00062) announces 2B / 14B configurations; it has a paper and does not belong in the "blog-only" tier; these configurations are not inherited by Cosmos 3.
• Cosmos 3 (arXiv 2606.02800) is omnimodal: language, image, video, audio, action. Its mixture-of-transformers means every layer holds both a set of autoregressive reasoner parameters and a set of diffusion generator parameters — it is not a sparse-FFN MoE, and not "one expert per modality". Per its paper §2.5 and the HF model cards, the sizes are Edge 4B / Nano 16B / Super 64B, all three released (Edge on 2026-07-20), under the OpenMDW-1.1 licence.

Atlas (World Labs official blog, 2026-09-01, blog-only) has to be described at the level of its interfaces: autoregressive diffusion conditioned on text, image, camera pose, depth and spatial context; the blog shows results of up to one minute at 1440p. Demo length is not evidence of real-time speed, and "camera controllable", "sparse reconstruction" and "robot contact dynamics" are three different kinds of evidence that cannot substitute for one another.

5.4　Driving, and Sora as a contrast

For driving, only the task difference is kept: GAIA-2 (Wayve 2025, arXiv 2503.20523) makes one concrete point — adding action, road layout and driving conditions as conditions gets much closer to the controllability that driving decisions need than unconditional video generation does. GAIA-3 currently has information only at the press-release level (blog-only), so it is mentioned only at the level of its release.

Sora / Sora 2 are a contrast, not a protagonist: their public material is high-level reporting, and their goal is the quality and usability of general video generation, which is not the same problem as the main thread of "the consequences of actions and decisions" (reasons for exclusion in §A.2). Equating "video generator" with "world model" is exactly the starting point of the definition fight in §7.4.

§6 Three roads in embodied AI

For a generative model to actually help a robot, it must cross the same gap: what it generates is pixels, what the policy needs is actions. There are three clear roads today.

Road one: video as the plan, actions by inference. UniPi (Du et al. 2023, arXiv 2302.00111) first generates the task as a video of "how it should be done", then converts the video into executable actions with something like inverse dynamics. The plan lives in pixel space, the landing happens in action space.

Road two: generate data, then supply action labels. DreamGen (NVIDIA 2025, arXiv 2505.12705) augments training data with generated video, but generated video has no action labels — it recovers pseudo-actions with a LAM or an inverse dynamics model (IDM) (the LAM route takes the continuous pre-quantisation embedding, not the discrete code), turning "video" into "policy training data". That step is the necessary bridge; without it, synthetic video is of no direct use for policy training.

Road three: the model itself is the environment. Genie Envisioner (arXiv 2508.05635) connects the world model to policy learning and simulated evaluation, so the model is both data source and evaluation arena. SIMA 2 (Google DeepMind 2025, arXiv 2512.04797) is an agent acting inside Genie-generated worlds — it shows "you can play inside a generated world", which does not mean real-world transfer has been established.

§7 Evaluation and the definition fight

7.1　Can photorealistic video prove an understanding of physics

Physics-IQ's design (Motamed et al. 2025, arXiv 2501.09038) must be remembered precisely: 66 physical scenes, 396 videos filmed for real; the model is given a conditioning frame or a conditioning video (depending on whether the model under test is image-to-video or video continuation) and asked to generate five seconds of continuation, which is then compared with the real continuation; the metrics are spatial IoU, spatio-temporal IoU, weighted spatial IoU and MSE. It measures conditional video continuation, not physics question answering.

It also measures visual realism by "can an MLLM tell real video from generated video". Across models, the correlation between realism and the physics metrics is $r=-0.46$, $p=0.249$ — no significant correlation was detected with this set of models and this sample size.

7.2　Two benchmarks, two questions

WorldModelBench (arXiv 2502.20694) measures instruction following plus detection of physics/common-sense violations — whether the generated content follows the instruction and whether it breaks basic common sense. WorldScore (arXiv 2504.00983) measures controllability, quality and dynamics. They cannot substitute for each other: the former asks "is it obedient, does it make common-sense errors", the latter asks "is it controllable, does it look good, does it move correctly".

7.3　What the two newer benchmarks each add

WorldArena 2.0's (arXiv 2605.17912) increment is visuo-tactile signals, policy optimisation inside the model, and a real robot platform — pushing evaluation from "watch the video" to "optimise a policy inside the model and verify it on real hardware". WorldRoamBench's (arXiv 2606.31672) increment is stability during interaction: whether action following, visual consistency, physical plausibility and memory retention can all be maintained over a long interaction.

Both papers reach conclusions of the form "no model meets the requirements across the board". That conclusion is confined to the models and protocols each of them tested, and is not a universal judgement about the whole field.

7.4　The definition fight: several proposals, not a consensus

• GLP critique (arXiv 2507.05169) criticises the route of treating generative video models directly as general-purpose world models, and argues that predictive capability and decision utility must be argued separately.
• Definition & Roadmap (arXiv 2607.06401) offers a definition and a roadmap proposal whose taxonomy uses a different set of roles (Renderer / Simulator / Planner, etc.). It is not the Predictor / Simulator / Evolver one — the easiest misattribution in this field.
• L0–L7 position (arXiv 2606.15032) proposes a grading over several crossing axes, not levels along a single line.
• Agentic World Modeling (arXiv 2604.22748) discusses world models inside an agent's decision loop, caring about "which step of the decision the model serves"; "Predictor / Simulator / Evolver" comes from this paper.

7.5　Two things to say fairly

The value of the generative route: visible detail may directly determine contact, collision and reward — DIAMOND is the experimental support for that argument. Also, diffusion generation is not "predicting one average future with an MSE"; it models the conditional distribution, and a sample gives one concrete, sharp future rather than a blurred superposition of several.

The value of the representation-prediction route: ignoring unpredictable detail can save a great deal of compute and concentrates prediction on quantities that are useful downstream. But it has a hard boundary in the other direction — a small representation loss does not mean the consequences of actions are predicted correctly. Distance in representation space is defined by the model itself, and there is no free equivalence between it and "will this action knock the cup over".

§8 The three lineages side by side

§9 Code Patterns

Three core patterns; the full experiments are in code/world_models_toy.py. All three share import torch, torch.nn as nn, torch.nn.functional as F.

9.1　One RSSM step + balanced KL

def cat_kl(probs_q, probs_p, eps=1e-8):
    """KL(q||p) for grouped categorical distributions, probs: [..., G, C].
    Sum over classes first, then over groups = the joint KL -- free bits truncates this quantity."""
    return (probs_q * ((probs_q + eps).log() - (probs_p + eps).log())).sum(-1).sum(-1)

class RSSM(nn.Module):
    def __init__(self, deter=512, groups=32, classes=32, a_dim=6, emb_dim=1024, hid=256):
        super().__init__()
        self.groups, self.classes = groups, classes
        z_dim = groups * classes
        self.cell = nn.GRUCell(z_dim + a_dim, deter)          # h_t = f(h_{t-1}, z_{t-1}, a_{t-1})
        self.prior_head = nn.Sequential(nn.Linear(deter, hid), nn.SiLU(), nn.Linear(hid, z_dim))
        self.post_head = nn.Sequential(nn.Linear(deter + emb_dim, hid), nn.SiLU(), nn.Linear(hid, z_dim))

    def _probs(self, logits, uniform_mix=0.01):
        logits = logits.unflatten(-1, (self.groups, self.classes))
        return (1 - uniform_mix) * F.softmax(logits, -1) + uniform_mix / self.classes   # 1% uniform mixing

    @staticmethod
    def _sample(probs):
        idx = torch.multinomial(probs.reshape(-1, probs.shape[-1]), 1).view(probs.shape[:-1])
        z = F.one_hot(idx, probs.shape[-1]).float()
        return z + probs - probs.detach()                     # straight-through: biased but differentiable

    def step(self, h_prev, z_prev, a_prev, emb_t=None):
        """emb_t=None is imagination: no observation, so only the prior is available."""
        h = self.cell(torch.cat([z_prev.flatten(-2), a_prev], -1), h_prev)
        prior = self._probs(self.prior_head(h))
        if emb_t is None:
            return h, self._sample(prior), prior, None
        post = self._probs(self.post_head(torch.cat([h, emb_t], -1)))
        return h, self._sample(post), prior, post

def balanced_kl(post, prior, beta_dyn=0.5, beta_rep=0.1, free_nats=1.0):
    """betas from DreamerV3 paper v1 Eq. 4 / Table W.1; the 32x32 categorical latent and free_nats=1 also follow the paper (Table W.1, Eq. 5)."""
    kl_dyn = cat_kl(post.detach(), prior)                     # sg(q)||p: only the prior chases the posterior
    kl_rep = cat_kl(post, prior.detach())                     # q||sg(p): only the posterior concedes
    return beta_dyn * kl_dyn.clamp(min=free_nats) + beta_rep * kl_rep.clamp(min=free_nats)

post.detach() stops the gradient on that path; $h_t$ is shared by the two heads, and its gradient still comes back along the other one.

In this code $\beta_{dyn},\beta_{rep}$ (v1 Eq. 4 / Table W.1), the 32-group × 32-class categorical latent (Table W.1) and free_nats=1.0 (the 1 nat of Eq. 5) all follow the DreamerV3 paper; using GRUCell as the recurrent cell and the network widths are choices made for readability, not the only implementation the paper prescribes.

9.2　JEPA: asymmetric predictor + EMA target

@torch.no_grad()
def ema_update(target, online, m=0.999):
    for p_t, p_o in zip(target.parameters(), online.parameters()):
        p_t.mul_(m).add_(p_o.detach(), alpha=1 - m)           # the target branch moves only by EMA

def jepa_loss(enc, enc_ema, pred, x_ctx, y_tgt, mask_tok, p=1):
    """p=2 -> I-JEPA's squared L2 on patch representations; p=1 -> V-JEPA / V-JEPA 2's L1."""
    ctx = enc(x_ctx)                                          # online branch, has gradients
    with torch.no_grad():
        tgt = enc_ema(y_tgt)                                  # stop-gradient: the target is not pulled toward the prediction
    out = pred(ctx, mask_tok)                                 # asymmetric: the predictor is on this side only
    return (out - tgt).abs().mean() if p == 1 else (out - tgt).pow(2).mean()

def latent_mpc_cost(pred_ac, s0, action_seq, z_goal):
    """V-JEPA 2-AC's planning interface: the encoder is frozen, only the action-conditioned predictor rolls."""
    s = s0
    for a in action_seq.unbind(1):                            # action_seq: [B, H, a_dim]
        s = pred_ac(s, a)
    return (s - z_goal).abs().flatten(1).mean(1)              # CEM ranks by this, replan after the first action

9.3　The VQ bottleneck of a latent action model

class LatentActionModel(nn.Module):
    def __init__(self, obs_dim, d=32, K=8):                   # Genie 1 uses 8 codes
        super().__init__()
        self.enc = nn.Sequential(nn.Linear(2 * obs_dim, 128), nn.SiLU(), nn.Linear(128, d))
        self.codebook = nn.Embedding(K, d)
        self.dec = nn.Sequential(nn.Linear(obs_dim + d, 128), nn.SiLU(), nn.Linear(128, obs_dim))

    def forward(self, o_t, o_next, beta_commit=0.25):
        e = self.enc(torch.cat([o_t, o_next], -1))            # E(o_t, o_{t+1}): the encoder can see the future
        d2 = (e.pow(2).sum(-1, keepdim=True) - 2 * e @ self.codebook.weight.t()
              + self.codebook.weight.pow(2).sum(-1))          # ||E(.) - e_j||^2
        k = d2.argmin(-1)
        e_k = self.codebook(k)
        e_st = e + (e_k - e).detach()                         # straight-through: the standard VQ formulation
        pred = self.dec(torch.cat([o_t, e_st], -1))           # the decoder gets only current information + the code
        loss = (F.mse_loss(pred, o_next)                      # prediction term
                + F.mse_loss(e_k, e.detach())                 # codebook term
                + beta_commit * F.mse_loss(e, e_k.detach()))  # commitment term
        return pred, k, loss

The bottleneck's entire job is to limit the amount of information: at $K=8$ one code is at most 3 bits. The decoder only gets $o_t$ and $e_k$, so $e_k$ has to carry "the bit of the inter-frame change most worth keeping" — but whether that is an action, the code does not guarantee (§5.1). The beta_commit=0.25 here is the common empirical value from VQ-VAE, copied over as a demonstration; Genie prescribes no such number.

§10 25 high-frequency interview questions

L1 essentials (10)

L2 advanced (10)

L3 top-lab (5)

§A Appendix

A.1　Papers and provenance

Evidence tier: ✅ paper (public on arXiv; the details cited are taken from the paper) / ⚠️ blog-only (official blog post or press release only, no paper). A third case is written out separately: a paper exists but only the direction is mentioned (its numbers were not individually verified), and it is not merged into "blog-only".

Lineage one: latent dynamics / MBRL

Lineage two: predict-in-representation-space

Lineage three, embodiment and evaluation

Openness: Cosmos 3's Edge, Nano and Super are all released (Edge released 2026-07-20, see its HF model card) under the OpenMDW-1.1 licence; Genie 3, Project Genie, Atlas and GAIA-3 have information only at the official blog level (Cosmos-Predict2.5 has a paper, arXiv 2511.00062). For the remaining entries the only assertion is that "the paper is public"; whether code, weights and data are open is governed by each project's own official page — no per-entry assertion is made, because those four things are frequently out of sync.

A.2　Exclusion list and reasons

A.3　Runnable toy

Script: code/world_models_toy.py, pure PyTorch, seconds on CPU. All three experiments have analytic answers you can check item by item — the conclusions cover only their own explicit settings and are not paper reproductions.

Experiment 1 | RSSM linear-Gaussian: posterior collapse and balanced KL. The environment is $x_{t+1}=0.5x_t+a_t+\epsilon$ with $\epsilon$ equally likely to be $\pm1$. The deterministic part $h=0.5x_t+a_t$ is computable exactly, and the residual $\epsilon=o_{t+1}-h$ is the only new information. With the scalar parameterisation $q(z\mid h,o)=\mathcal N(w\epsilon,1)$, $p(z\mid h)=\mathcal N(v,1)$, $\hat o=h+dz$, we get $\mathcal L_{rec}=\tfrac12[(1-dw)^2+d^2]$ and $\mathbb E D_{KL}=\tfrac12(w^2+v^2)$. Comparing a single KL coefficient of 9 with balanced $\beta_{dyn}=9,\beta_{rep}=0.1$ (toy values, not the paper's): starting from $w=d=0.5$, $v=1$ and running SGD at lr 0.05 for 1,000 steps, the single-coefficient version converges to $\lvert w\rvert,\lvert d\rvert<10^{-6}$ — posterior collapse, with reconstruction MSE ≈ 1, i.e. the observation is not used at all; the balanced version converges to $v\to0$, $w^2=1/\sqrt{0.1}-1$, $d=w/(1+w^2)$, with MSE $=\sqrt{0.1}$.

Experiment 2 | JEPA's shortcut and stop-gradient. With $x=s$, $y=s+\epsilon$, an encoder $E_w(u)=wu$ and an identity predictor: sharing $w$ and not stopping the gradient gives $\mathcal L=\tfrac12w^2$, and gradient descent at lr 0.1 gives $w_n=0.9^n$ — starting from $w_0=1$ and running 100 steps, the context representation's variance falls to $0.9^{200}$; collapse is an attractor. Switching to stop-gradient + an EMA teacher $b\leftarrow0.9b+0.1w$: starting from $w=b=1$ the gradient is zero, $w$ stays at 1, and the loss of 0.5 comes entirely from unpredictable noise; but starting from $w=b=0$ it stays put just the same. This is the executable version of "a mechanism is not a theorem".

Experiment 3 | A coordinate LAM on a grid world. The encoder takes $\Delta=s'-s$, the VQ has 4 codes, and the decoder is additive. After farthest-first initialisation plus one round of nearest-neighbour assignment and centroid update, the four transition vectors are recovered exactly: validation reconstruction error 0, purity 1, $I(K;A)=2$ bits (up to a permutation of the codes). This shows that in this coordinatised setting with no exogenous change the bottleneck does capture the action — once high-variance change unrelated to the action is added, the competition described in §5.1 begins.