Reading, Writing, Sharing, and Interacting with the World through Video
Video foundation models now handle both understanding and generation of dynamic visual content, but the field lacks a principled way to relate these tasks to an underlying physical world. This survey proposes a world-modeling lens that partitions the literature into four operations—reading, writing, sharing, and interacting—each connecting observed video traces to a latent temporal world. By distinguishing mere temporal coherence in generated sequences from true consistency of the underlying world state, the work reframes evaluation criteria and identifies open challenges in building unified video world models.
The survey delivers a needed conceptual architecture for a fragmented field. Its formal definitions of latent world state $W$, observation process $\mathcal{O}$, and world consistency (Definition 3.4) convert intuitive desiderata into precise taxonomic axes, while the evidence ladder (Table 2) disciplines claims by requiring that "Each operation can be supported at four increasing levels; stronger world-modeling claims require the higher levels." At the same time, the sheer breadth—hundreds of papers across four major sections—means the synthesis relies heavily on authors' interpretive grouping, and the 2026 preprint date raises concerns about citation currency and peer review status.
The core distinction between temporal coherence and world consistency is carefully articulated: "An object that changes identity after leaving the frame, a room that alters its layout between visits, or an action whose effect later disappears can yield locally smooth frames that no trajectory in $\mathcal{C}$ explains; conversely, a hard cut violates local smoothness while remaining world-consistent" (Section 3.2). The two-axis taxonomy of video world models by prediction space and transition interface (Section 4.4) is likewise a clean analytical tool that maps neatly onto existing work, from observation-space rollout models to latent-state planners.
The survey covers hundreds of recent preprints—many dated 2025 and 2026—making some classifications inevitably provisional. The authors acknowledge conceptual leakage themselves: "The arrows here are conceptual shorthand: writing reaches $W$ only through $c$, and reading and writing are not exact inverses" (Section 3.3), which blurs boundaries in practice. While the survey rightly warns that "Aggregate accuracy does not show that a model used the intended evidence" (Section 5.1), it does not provide a systematic selection protocol, leaving the review vulnerable to coverage gaps in explicitly excluded areas such as 3D/4D geometric world models (Section 4.4).
The survey's evidence is qualitative synthesis rather than new experiments, which is appropriate for the genre. Claims of convergence—"These two lines of progress are increasingly converging around a common challenge: connecting video observations with the dynamics of the world they depict" (Section 2)—are stated rather than demonstrated through quantitative meta-analysis. Comparisons to prior surveys are candid about differing scope, positioning this work as distinct in its world-modeling focus rather than merely broader.
Reproducibility for a survey depends on transparent methodology and available resources. The authors publish a project page and GitHub repository, and compile extensive tables of tasks, datasets, and benchmarks. However, no explicit search strategy or inclusion protocol is stated, and the authors note they are "avoiding assumptions that strong performance on individual video tasks necessarily implies consistent or generalizable world modeling" (Introduction), yet how each model was classified into the prediction-space versus transition-interface grid is not independently reproducible from the text alone. The heavy reliance on partially described systems means many entries rest on public reports rather than inspected artifacts.
Video foundation models have advanced rapidly in understanding and generating visual sequences, while video world models seek to capture the underlying dynamics of the world they depict. Despite their different objectives, both concern the relationship between video observations and an evolving world. This survey adopts world modeling as a unifying lens for examining these developments. Viewing an observed video as a partial, lossy, and temporally ordered projection of a latent temporal world, we organize the literature around four complementary operations: reading infers world properties from observed video; writing synthesizes video from specified or learned world conditions; sharing integrates understanding and generation through common modeling mechanisms; and interacting predicts and updates world states under actions and evolving observations. This perspective distinguishes temporal coherence in observed or generated sequences from consistency in the underlying world state. Across tasks, representations, models, datasets, and benchmarks, we trace a shift in emphasis from clip-level recognition and perceptual fidelity toward persistent state, controllable evolution, physical and causal validity, and evaluation grounded in interaction and downstream decisions, and we distill the open challenges that follow.
Pick a starting point or write your own. Challenges run in the background, so you can keep reading while the AI investigates.
No challenges yet. Disagree with the review? Ask the AI to revisit a specific claim.