In CEM, a World Model Is Also a Proposal Mechanism
In model-predictive control with the cross-entropy method, a learned world model scores candidate action sequences and refits the Gaussian proposal for the next search iteration, creating planner-induced query shift. The authors disentangle these two roles by saving every CEM candidate pool from four surrogate models, executing all candidates in a resettable environment for a ground-truth ranking, and rescoring each pool with every model in a crossed design; they then splice the first elite set with environment-ranked elites to trace causal consequences through later iterations. Their results show that standard proposal distances contract even as ranking agreement stays near chance and mean disagreement grows, and that a single environment-guided elite update lowers final selected cost—demonstrating that world-model quality must be judged by its effect on the proposal, not just on candidate scoring.
The paper presents a careful, pre-registered audit of CEM planning that demonstrates a world model's scoring errors propagate through the proposal distribution, not just the immediate selection. The crossed design cleanly separates pool generation from scoring, and the held-out one-update intervention shows that replacing the first model-ranked elite set with an environment-ranked one lowers final realised selected-sequence cost in all twelve task-seed units. However, the scope is narrow—two tasks, one horizon, one CEM configuration—and the intervention tests only one surrogate family selected by a descriptive residual pattern, leaving open whether the finding generalises to other model classes or planning settings.
The experimental design is the paper's strongest feature. By saving pools and reusing environment executions across all generator-scorer pairs, the crossed matrix isolates disagreement due to pool source, scorer, and their interaction without requiring new simulator rollouts for off-diagonal cells. The authors also preregistered their selection rule and tested the resulting intervention on held-out seeds without inspecting outcomes, guarding against cherry-picking. Their metric decomposition is equally careful: separating $D_\mu$ and $D_{\log\sigma}$ reveals that a falling aggregate distance $D$ can mask rising mean disagreement, a nuance lost in a single score.
The descriptive crossed matrix supports association, not causation, and while the authors note that it 'does not assign causal effects to those components,' the intervention is performed only on Random nonlinear, selected because it showed a positive own-pool residual in the pilot. It remains unknown whether the other three families would have benefited equally or more from the same splice. More fundamentally, the experiment covers two tasks, a fixed fifteen-step horizon, and one CEM configuration; the Identity surrogate also retains a 1280-dimensional context while the compressed families use 256, confounding feature-map differences with dimensionality. The splice is an oracle diagnostic requiring environment executions for every candidate, and the authors correctly note that lower finite-horizon selected-sequence cost does not establish higher closed-loop return.
The evidence directly supports the central descriptive claims. The contraction of $D$ is driven by falling width disagreement, mean separation rises in current-proposal units, pairwise order agreement stays near chance on Walker and declines on Cheetah, and Cheetah matrices are scorer-dominated while Walker shows mixed variation. Comparisons to prior work are fair: the authors correctly distinguish their fixed-pool crossed audit from Wang et al.'s trace-only Spearman analysis and from Li et al.'s adjacent-set reconstruction, noting that their intervention changes which pools are sampled next. Still, the leap from a single-family oracle splice to the general thesis that the world model is 'also a proposal mechanism' is largely conceptual; the experiment validates existence but not ubiquity.
Reproducibility is mixed. On one hand, the paper provides exhaustive hyperparameters in Appendix Table 9, including Dreamer architecture, surrogate MLP shapes, readout settings, loss weights, and optimisation details, and it estimates compute at approximately 179 GPU-hours. On the other hand, the code, trained checkpoints, and saved simulator states are not yet public, which currently blocks independent reproduction. The use of frozen backbones, fixed base random samples, and matched noise branches nonetheless means the pipeline is deterministic and should be reproducible once the release is completed.
The cross-entropy method (CEM) uses world-model scores to select action sequences and fit the distribution sampled in its next iteration. A scoring error can therefore change both the present decision and the candidates considered later. We evaluate these two roles separately. Four types of predictive model generate CEM traces, and every model rescores every saved candidate pool. Executing the same candidates in the environment provides a reference elite set and proposal update. Across twelve independently trained task-seed units on Walker and Cheetah, the pre-specified proposal distance falls from the first to the final CEM iteration in every unit. Proposal widths contract and fitted means separate relative to the remaining search width. Pairwise ranking agreement stays near chance on Walker and declines on Cheetah; elite-set agreement does not improve. This comparison shows greater variation between scorers than between pool sources on Cheetah; Walker has variation in both and in their pairings. We use the original six units to select Random nonlinear for a one-update intervention, without inspecting intervention outcomes. Replacing its first model-ranked update with an environment-ranked update lowers final realised selected-sequence cost in those six units and in six further units held out from the selection.
Pick a starting point or write your own. Challenges run in the background, so you can keep reading while the AI investigates.
No challenges yet. Disagree with the review? Ask the AI to revisit a specific claim.