Calibration-risk routing for controlled world-model adaptation
Model-based reinforcement learning can leverage simulators to avoid risky target interactions, but dynamics shifts create a dilemma: should one correct the simulator or fit the target directly from limited data? This paper introduces MC-WM, which trains both a residual-corrected simulator and a direct target world model, then selects between them using a held-out calibration-risk score. The core value is a disciplined experimental protocol—including exact routing counterfactuals and careful access accounting—that tests whether calibrated model-family selection outperforms fixed strategies.
The paper presents a carefully controlled study of model-family selection under simulator-to-target shifts, and its central claim is appropriately bounded. The evidence supports that calibration-risk routing improves over fixed residual deployment and model-free baselines, though it does not establish superiority over direct target modeling. As the authors note, "The supported claim is therefore calibrated choice under these shifts, not universal routing or model-based dominance."
The experimental rigor is the paper's greatest strength. The exact routing counterfactual—where automatic routing must byte-identically reproduce the selected forced arm—rules out confounding from initialization or random streams. The authors state that "The automatic arm must exactly reproduce its selected forced arm; Appendix C.1 details this isolation and matching test." Additionally, the honest reporting of unresolved comparisons and the explicit separation of training, validation, and test access demonstrate uncommon scientific discipline.
The main limitation is narrow scope: only three controlled MuJoCo simulator-to-simulator shifts are tested with ten seeds each, and the strongest model-based comparator—direct target MBPO—shows no resolved return difference on any environment. The authors correctly report that "return differences from direct target MBPO remain unresolved," meaning the benefits of the full residual-corrected pipeline over simply fitting a target ensemble remain unproven. Moreover, the symbolic residual shows very low cross-seed support Jaccard overlap (0.20–0.26), undermining claims of stable structural discovery; the authors concede that "stable structural discovery and sparse control superiority do not" hold.
The routing return benefit is also only resolved on Ant, as gravity and Walker automatic-minus-alternative CIs include zero.
The evidence supports selection over fixed residuals and over DARC and target SAC, but comparisons to direct target MBPO are inconclusive. The family selector minimizes the standardized calibration risk $\mathcal{R}_{\mathfrak{m}}$ in Equation (3), yet this one-step transition metric does not guarantee downstream control performance. The selector chooses direct models for Ant and gravity and residual for Walker, but only the Ant routing contrast shows a resolved return improvement ($1529.9$ $[426.9, 2833.1]$). The authors write: "Ant supplies a resolved route-return benefit; gravity and Walker return effects and all route-level violation effects are unresolved." The sparse-versus-dense study finds fewer parameters but no endpoint advantage, weakening the case for the sparse-plus-neural backend. The DARC comparison is further tempered by a scaled warmup the authors note is "not presented as a reproduction of DARC's original longer schedule."
Reproducibility appears exceptionally thorough. The protocol fixes random-stream isolation for direct-candidate fitting, freezes pre-deployment candidates, and reports 540 unique cells from 541 executions with source-tree hashes. Hyperparameters are tabulated in Table 9, and Algorithm 1 gives the complete training loop. The authors state that "Appendix G identifies the frozen source and the single command that regenerates the validated submission outputs." The one replaced cell was pre-registered before endpoint inspection. The only potential barrier to independent reproduction is the temporary anonymous-supplemental archive; long-term availability depends on the promised public repository.
Model-based reinforcement learning (MBRL) can exploit simulated experience, but a simulator-to-target shift creates a model-selection problem: correcting the simulator and fitting the target directly can each fail under limited target data. We introduce the Model-Corrected World Model (MC-WM), which separates initial target data into disjoint fit, selection, and calibration partitions and deploys the family with lower standardized calibration risk. A learned confidence signal and deterministic validity predicates weight one-step imagined policy updates without rewriting physical rewards. We evaluate 540 unique reported run cells across three controlled Multi-Joint dynamics with Contact (MuJoCo) shifts; one exact-routing cell was repeated after a pre-deployment artifact gate, giving 541 completed executions.
Pick a starting point or write your own. Challenges run in the background, so you can keep reading while the AI investigates.
No challenges yet. Disagree with the review? Ask the AI to revisit a specific claim.