Generative Representation Learning for Scalable Latent Tokenization
Training good latent tokenizers for generative models remains poorly understood, especially how to scale the encoder itself. This paper proposes Relative Prediction Representations (RePR), which supervise patch features to factorize a pointwise mutual information matrix estimated by a conditional diffusion model. The result is a theoretically grounded, negative-sample-free objective that resists feature suppression and claims to be the first to demonstrate compute-optimal tokenizer encoder scaling, matching or beating pixel-space diffusion at equal FLOPs.
The paper offers an elegant theoretical reframing—explicitly estimating PMI with a generative model to supervise patch features—and provides encouraging empirical scaling curves. Yet the boldest claims are projections: the authors state, 'Extrapolating the observed trend by at most 1× beyond our measured log-scale FLOP range predicts DINOv2-level performance at approximately one-seventh the training FLOPs and WebSSL-level performance at roughly one-eighth' (Section 6.2). They also report that 'At a compute budget of $1.19 \times 10^{20}$ FLOPs, equal to 200 epochs of JiT-L training, RePR with JiT-L improves unguided FID-10K by 2.61 for conditional generation and 4.18 for unconditional generation versus JiT-L on pixels' (Section 6.3). These results are promising but remain tied to the authors' Latent Forcing pipeline and to idealized theoretical assumptions.
RePR's PMI factorization is theoretically elegant: it directly supervises dot products of patch embeddings to match a generative PMI estimate, inheriting the guarantees of contrastive learning without negative samples. The watermarking experiment is especially clean: 'On watermarked data, LeJEPA and DINOv3 collapse to only representing the shared watermark. Meanwhile, despite sharing the same PMI factorization guarantees as contrastive learners, our proposed RePR is robust' (Section 6.1). An independent evaluation of the same watermarking protocol confirms that 'This old failure mode is easily reproduced in every modern JEPA-style system we tested (DINOv3, LeJEPA, and LeWM), despite each system's built-in anti-collapse defenses' (Enigma blog).
The headline 7× and 8× FLOP savings are extrapolations from power-law fits rather than measured results. The authors write, 'Extrapolating the observed trend by at most 1× beyond our measured log-scale FLOP range predicts DINOv2-level performance at approximately one-seventh the training FLOPs and WebSSL-level performance at roughly one-eighth' (Section 6.2). This makes the compute-optimal claim speculative, especially because the width-1024 model begins to overfit after 200 epochs on ImageNet (Section 6.2).
In addition, the theoretical justification assumes an idealized denoiser: the authors note that '$\log p(\mathbf{x}) = \text{VLB}(\mathbf{x})$, when the denoising network is optimal and the endpoint terms vanish' (Section 4.1). The headline FID-50K of 2.18 is also achieved inside the authors' own Latent Forcing framework, a specialized decoder-free pipeline that is not yet part of standard latent-diffusion recipes, limiting the generality of the headline result.
Within the paper's own experimental framework the evidence is largely consistent, but comparisons to prior work are uneven. The authors acknowledge that 'RAE and REPA do not control for representation learner pretraining, where the chosen latent space, DINOv2-b (Oquab et al., 2024) is distilled from a 1.1 billion parameter model trained on over 100 times more data than ImageNet' (Section 2). They further note that 'The RAE decoder relies on two external models, LPIPS for a perceptual loss, and a DINOv1/8 pretrained GAN that uses $4.7 \times 10^{20}$ FLOPs, more than any model we train' (Section 6.5). These admissions underscore that prior tokenizer baselines carry large hidden compute costs, but they also mean the playing field is not level.
The authors release code and provide detailed hyperparameters. The abstract states, 'Code at https://github.com/AlanBaade/repr' (Abstract). Nevertheless, the full pipeline is nontrivial: it requires iterative feature prediction, a custom MAR-based prediction model, and the Latent Forcing decoder. The authors estimate that 'the results in Fig. 3 could be replicated on a single 8 GPU B200 node in 250 hours' (Appendix A.3), but verifying the headline FID-50K of 2.18 demands retraining both the tokenizer and a JiT-L generative model, a substantial compute commitment.
We introduce an approach for self-supervised representation learning designed to be used as a tokenizer for generative modeling. Our objective supervises a representation loss by factorizing the pointwise mutual information of the data, which we estimate by taking information differences between conditional predictions from a learned generative model. We justify our algorithm theoretically and empirically, demonstrating that our generative approach to representation learning avoids dimensional collapse and feature suppression in cases where commonly used methods like DINO and LeJEPA fail. Moreover, we show our representation learning technique is scalable; expending more compute leads to predictably better latent tokenization. For the first time, we perform a careful analysis of the combined flop efficiency of tokenizer and generative model and demonstrate that our approach is a compute-optimal improvement over direct diffusion for pixel-space generation, obtaining an FID-50K of 2.18 with JiT-L. Applied to ImageNet and the text-to-image dataset GPIC, we show tokenizer encoder scaling laws for diffusability, where extrapolating our approach achieves representations in 7x fewer FLOPs than DINOv2 on ImageNet and 8x fewer FLOPs than WebSSL on GPIC.
Pick a starting point or write your own. Challenges run in the background, so you can keep reading while the AI investigates.
No challenges yet. Disagree with the review? Ask the AI to revisit a specific claim.