Inside-JEPA: From Prediction to Relational Grounding in Latent World Models
Inside-JEPA tackles the opacity of latent world models by embedding relational channels for ordered entity pairs directly into the JEPA latent state. The core idea is that an inverse-dynamics loss—recovering actions from changes in these channels—can ground them to geometric relations without any relation labels during training. This matters because it could let world-model states be read and given goals in relational terms, the natural vocabulary for planning and manipulation.
The paper presents a carefully controlled empirical study showing that latent prediction alone fails to give meaning to dedicated relational channels, while inverse-dynamics supervision that reads those channels produces consistent but modest grounding gains across ten annotated-box settings, robot episodes, and tracked objects. The evidence is presented transparently alongside extensive negative results, though the effect sizes remain small relative to a supervised ceiling and the experimental scope is deliberately narrow.
The controlled arm design is the paper's strongest feature. By sharing architecture, data, budget, and loss weight across variants and varying only whether the inverse head reads the relational channels, the entity latents, or the full state, the authors cleanly isolate the cause of grounding. The finding replicates across Blocks2D, Something-Something V2 with annotated or tracked objects, and RH20T robot episodes; INV-S consistently outperforms PRED, while shuffled labels and label-free controls fail.
The most striking failure is that left/right relations never ground on real video, despite being easily decoded by a linear probe from the same latent, and the five tested explanations do not account for it.
Robot results rely entirely on human-annotated boxes because automatic detection and tracking failed quality checks, and the gain with tracked objects is roughly half that of the annotated-box setting. Planning is limited to one-step action selection using readouts fitted on relation-labelled pairs, and the paper deliberately avoids comparing against prior relational world models. Absolute gains are modest—typically .02–.10 AUROC—and a stronger encoder narrows the gap, suggesting sensitivity to the feature backbone.
The internal comparisons are rigorous and fair: every arm shares the same architecture, data, budget, and hyperparameters, and the random-head replacement test guards against spurious post-hoc matching. However, the paper does not benchmark against prior relational world models like C-SWM or slot-based rollouts, and it substitutes INV-Z as a proxy for recent whole-latent inverse-dynamics terms rather than running those methods directly. The relation labels used only for evaluation are geometric proxies computed from 2D boxes, not physical relations, which weakens the claim that the channels represent physical grounding beyond box geometry.
Hyperparameters, architecture details, and training budgets are documented in Appendix A with unusual precision, including exact step counts, optimizer settings, and momentum coefficients. The datasets are public, and the evaluation protocol is pre-registered with locked decision rules described in Appendix K. What would block reproduction is the absence of any mention of released code or pretrained checkpoints, the reliance on specific annotated box sets and tracker settings selected on a development sample, and the small model size that may hide sensitivity to initialization across the three to five seeds reported.
Joint-embedding predictive architectures (JEPAs) learn world models in latent space, enabling efficient planning but leaving their states opaque. We introduce Inside-JEPA, which places relational channels for ordered entity pairs inside the JEPA latent state and shows that an inverse-dynamics loss can ground these channels without relation labels when its head reads them, either alone or as part of the full state. On Something-Something V2, latent prediction alone performs no better than a random relation head, while inverse-dynamics supervision that recovers actions from channel changes improves relational AUROC by .065-.103 across ten settings with annotated boxes, by .081 on RH20T robot episodes with held-out tasks, and by .050-.061 across five relations using off-the-shelf detection and tracking. An inverse head on entity latents decodes actions more accurately but barely improves relational grounding; shuffled labels, two label-free objectives, and action labels on only 5% of the pairs also fail to ground the channels. Given an action, predicted relational changes approach the quality of observed future relations (AUROC .599 vs .621). For one-step relational goals, a supervised readout of the 16 channels ranks the correct action among the top five of 65 in 43% of cases, compared with 24% for prediction-only latents and 17% for a frequency prior.
Pick a starting point or write your own. Challenges run in the background, so you can keep reading while the AI investigates.
No challenges yet. Disagree with the review? Ask the AI to revisit a specific claim.