SceneAE: Scene Generation and Reconstruction in Queryable 3D-Anchored Latent Space
SceneAE proposes a sparse 3D-anchored latent representation for scene reconstruction and camera-controllable generation. Instead of viewpoint-aligned video latents, each latent couples an explicit 3D point with an implicit feature, enabling queryable rendering from arbitrary cameras. A two-stage diffusion model decouples geometry generation from content synthesis, improving camera control and cutting computational cost roughly in half compared with dense-grid baselines.
The paper presents a compelling idea with solid empirical results, but some claims are overstated and comparisons are partially uneven. The core technical contributions — explicit 3D anchor positions and a geometry-content decoupled diffusion transformer — are well-motivated and improve multi-view consistency and trajectory following. However, the reconstruction fidelity remains below dense video-VAE baselines, and the method’s reliance on static-scene assumptions limits applicability.
The strongest aspects are the design of the latent point cloud and the splat-based decoder, which together enable efficient queryable reconstruction from arbitrary viewpoints. The ablation in Table 9 shows that decoupling geometry and content yields clear gains, and the paper demonstrates favorable efficiency: with only $N=3000$ latents, GFLOPs and latency are roughly half those of SceneTok.
Key flaws include lower reconstruction fidelity compared to dense latents, which the authors acknowledge in Appendix C.2 but downplay in the main text. The evaluation of geometry generation uses different ground-truth sources for different methods — VGGT for Gen3R and DA3 for the authors’ method and GLD — making the comparison unfair and potentially inflating the reported Chamfer-distance gap. The method assumes static scenes, so the generation task is essentially static-scene extrapolation rather than true dynamic world modeling. Additionally, the two-stage training and inference pipelines increase complexity, yet the paper does not report end-to-end training time or sensitivity to the fixed latent budget.
The evidence broadly supports the central claims about geometry consistency and camera controllability, but the comparison geometry is uneven. Metrics in Table 2 are computed against different depth estimators per method, preventing a fair ranking. The appearance metrics in Table 1 show competitive PSNR and SSIM, yet the gap over GLD is small. Novel-view synthesis beats SceneTok and LVSM but still trails dense methods that do not support novel-view rendering.
Training details are relatively thorough: DA3-Base features, $N=3000$ latents, $518 \times 376$ resolution, AdamW with learning rate $5 \times 10^{-5}$, batch size $40$, and $100$k iterations on 8 A800 GPUs. However, no code, data splits, or pretrained weights are released. The reliance on a frozen DA3 model and specific ray tokenization means reproduction requires exact feature-extraction pipelines that are not fully specified. The jitter, dual-path deformable attention, and two-stage designs introduce many hyperparameters that are only partly documented.
Recent video world models have increasingly adopted 3D memory mechanisms to integrate historical observations into a lightweight and persistent scene representation. This naturally raises a complementary question: if 3D representations can serve as an effective mechanism for memorizing observed scenes, can they also be directly used as the latent space for reconstructing and generating scenes? Motivated by this idea, we represent a scene as a compact and unstructured 3D-anchored latent space (\textbf{SceneAE}). Each latent combines an explicit 3D position with an implicit content feature and is spatially grounded to describe its local scene region, forming a sparse latent point cloud. Unlike conventional video-VAE latents that describe how a scene appears from particular viewpoints, our representation directly describes the scene itself, making it naturally applicable to both reconstruction and generation. For reconstruction, once the scene has been generated, our persistent 3D representation can be projected and decoded into RGB-D observations by different target cameras, without the need to regenerate a video for each new camera trajectory. For generation, we leverage the geometry-content separation to first generate camera-controlled 3D geometry and then synthesize content conditioned on it, reducing camera-appearance entanglement and improving camera controllability. Experiments show that this 3D latent achieves superior generation quality, computational cost, and camera controllability.
Pick a starting point or write your own. Challenges run in the background, so you can keep reading while the AI investigates.
No challenges yet. Disagree with the review? Ask the AI to revisit a specific claim.