Learning from Next-Frame Prediction: Towards Unified and Scalable Visual Representation Learning
Most video pretraining methods recover masked content within a clip rather than predicting the future, and autoregressive token prediction has lagged joint-embedding methods in representation quality. VANE proposes next-frame prediction in a frozen semantic embedding space, using a causal encoder and a lightweight generative decoder to preserve spatial detail, and extends the same objective to still images via local-to-global crops. This unifies image and video pretraining under a single scalable objective centered on temporal evolution, yielding strong gains on anticipation, recognition, and dense benchmarks.
VANE is a conceptually clean and empirically strong contribution to self-supervised visual pretraining. By shifting from masked in-filling to next-frame prediction in embedding space, it achieves sizable gains on anticipation and dense perception benchmarks and shows consistent scaling with model size and data. The matched ablation against V-JEPA 2.1 under the same data and target space validates that the objective itself drives much of the benefit. That said, the headline comparisons to prior work are based on 1B-scale baselines; the V-JEPA 2.1 paper reports higher scores with scaled-up variants, so the state-of-the-art claim should be read with scale in mind. The paper would also be stronger with an analysis of failure modes and of the tokenizer's distribution bias.
The clearest strength is the controlled ablation showing that next-frame prediction outperforms next-token prediction in the same frozen DINOv2 target space by 19.1 points on SSv2 and 22.7 mIoU on ADE20K, which isolates the architectural objective from the choice of tokenizer. The local-to-global image extension is elegant and empirically justified, with next-crop prediction improving over masked-image completion by 6.8 mIoU. Finally, the auxiliary flow-matching decoder regularizes predicted states via $\mathcal{L}_{\mathrm{flow}}$ and yields measurable gains on dense tasks, confirming its role in preserving fine-grained spatial structure.
The paper presents its results against 1B-parameter baselines, yet the cited V-JEPA 2.1 work advertises higher scores on larger models—40.8 Recall@5 on EPIC-KITCHENS, 7.71 mAP All on Ego4D, and 0.307 RMSE on NYUv2—raising the question of whether VANE's margins are as large at matched 2B scales. The manuscript also lacks any analysis of failure modes or of the limitations imposed by freezing a DINOv2 tokenizer, whose semantic space may carry biases or blind spots that the predictor cannot overcome. Additionally, while the cumulative training recipe improves results, the relative contributions of architectural novelty versus scaling, longer temporal context, and higher resolution are entangled, and the gains attributed to next-frame prediction alone are smaller on some benchmarks (e.g., 0.2–0.3 points on IN1K/SSv2 from the generative decoder).
The evidence supports the central claim that next-frame prediction is a better pretraining objective than masked completion, especially under matched pretraining: when V-JEPA 2.1 is trained on the same VISTA corpus with the same frozen DINOv2 target, VANE still leads by 1.8 points on IN1K and 3.6 mIoU on ADE20K. Comparisons to Toto and VideoMAE are largely fair and draw from published numbers. However, the comparison to V-JEPA 2.1 in the main tables does not clearly flag that the larger 2B V-JEPA 2.1 model outperforms the 1B baseline shown, which blurs the interpretation of state-of-the-art among unified encoders. The matched pretraining ablation in Appendix D.1 is the most rigorous evidence and should arguably be highlighted more prominently.
The authors provide extensive architectural details, hyperparameters, and a four-stage training schedule in the appendices, but no code or pretrained weights are released. The VISTA dataset is curated from public sources plus additional retrieved and class-balanced data whose exact construction is not fully specified, which would block exact reproduction. The total compute budget is large (~57k GPU-hours), and the reliance on a frozen DINOv2 tokenizer means reproducibility also hinges on that specific pretrained checkpoint. Standard downstream protocols are followed, but independent replication would require rebuilding the custom data pipeline and exact multi-stage schedule.
Video provides a natural source of supervision: what happens next serves as a learning signal for visual world modeling. Yet masked video pretraining focuses on recovering missing content from surrounding context rather than predicting the future from the past. Autoregressive methods instead predict forward one token at a time, but have yet to match joint-embedding methods in representation quality. This gap motivates us to revisit the granularity and space of future prediction. In this work, we introduce VANE, a self-supervised visual pretraining framework built on next-frame prediction in embedding space. A causal encoder summarizes past observations for a state predictor that infers the representation of the entire next frame, making temporal evolution the primary learning signal. This prediction is carried out in a fixed embedding space provided by a frozen semantic tokenizer, emphasizing scene content and structure. To retain fine-grained spatial detail alongside semantic content, a lightweight auxiliary generative decoder reconstructs future-frame patches conditioned on the predicted representations. To unify visual modeling across images and videos, we construct a local-to-global crop sequence for still images, allowing the same encoder to predict newly revealed regions from earlier crops under the same objective. Experiments show that VANE improves with model and data scaling, achieving SOTA on most anticipation, recognition, and dense perception benchmarks among unified encoders. Especially on EPIC-KITCHENS-100 action anticipation, it surpasses V-JEPA 2.1 by 7.1 points in mean-class recall@5. These results demonstrate that next-frame prediction in embedding space is effective and scalable for visual world modeling.
Pick a starting point or write your own. Challenges run in the background, so you can keep reading while the AI investigates.
No challenges yet. Disagree with the review? Ask the AI to revisit a specific claim.