LiFi: LiDAR Generation from Multi-View Images via Geometric and Semantic Collaborative Guidance
LiFi targets the task of generating metric LiDAR point clouds from cheap onboard multi-view camera images for autonomous driving. The core idea is a latent diffusion model guided by two decoupled branches: a geometry branch that aligns dense monocular depth estimates to the target LiDAR scanning pattern via a reverse sampling algorithm and uncertainty-modulated encoding, and a semantic branch that aggregates global cross-view context. If the approach holds up, it could enable large-scale data augmentation, cross-modal simulation, and sensor completion without requiring expensive LiDAR capture at deployment time.
LiFi offers a well-engineered and largely convincing solution to camera-to-LiDAR generation. The reverse sampling algorithm, uncertainty-aware FiLM modulation, and dual-branch balanced CFG are novel and well-motivated. Quantitative improvements over baselines are substantial: on nuScenes it achieves MMD 2.78 and JSD 3.66, outperforming camera-to-LiDAR rivals LiDM (4.57, 10.61) and X-Drive (48.58, 10.54) by large margins. However, the ablation reveals that RGB conditioning contributes almost nothing to MMD (2.80 vs 2.78), suggesting some components add complexity without commensurate fidelity gains. The downstream detection experiment is carefully caveated but still overclaimed as evidence of practical value when the authors admit it does not show independent support for perception. Overall, the paper makes a solid contribution despite slightly overstated novelty claims.
The geometry branch is the strongest part of the paper. The range-view-centered reverse sampling reduces complexity from $\mathcal{O}(H_{img} \cdot W_{img})$ to $\mathcal{O}(H_r \cdot W_r)$ by querying the LiDAR grid and sampling depth from camera views, which directly addresses the mismatch between dense images and sparse LiDAR patterns. The uncertainty-aware encoder uses FiLM to dynamically scale and shift features based on local depth confidence, a principled way to suppress unreliable geometry. The dual-branch balanced CFG is also a sensible extension of standard classifier-free guidance that explicitly prevents the geometry branch from overshadowing the semantic branch during training.
There are three primary weaknesses. First, the main-text ablation (Table 6) shows that dropping the RGB image condition barely hurts MMD (2.80 versus 2.78 with all modules), and removing the entire geometry-aware branch only raises MMD to 5.20 while JSD stays comparable (4.27 versus 3.66). This suggests the semantic branch carries most of the fidelity load, and the geometric innovations add engineering complexity with diminishing metric returns. Second, the range image representation is admitted to be lossy, and the long-range analysis shows severe degradation: CD grows from 0.11 within 10 m to over 32 beyond 40 m. Third, the downstream 3D detection result is presented as evidence of practical value, yet the authors explicitly note that it does not directly demonstrate independent support for real-LiDAR-comparable perception, undercutting the claim.
The evidence supports the core claim that LiFi outperforms existing camera-to-LiDAR methods. The comparison with X-Drive is fair and verified by the source paper, which describes X-Drive as a joint-generation framework using text prompts and 3D bounding boxes rather than an image-only pipeline. LiFi's margins on point-level metrics are decisive: CD 0.83 versus LiDM's 4.57 and X-Drive's 4.73. However, Table 1 conflates incomparable settings by pitting LiFi against unconditional and occupancy-conditioned generators under the same headers; these baselines do not accept camera images and therefore do not address the same task. The qualitative results are persuasive but lack explicit failure-case visualizations in the main text.
Reproducibility is decent but incomplete. Training details are specific: 200 epochs, 176,000 steps, learning rate $1\times 10^{-4}$, DDIM sampler with $T=1024$, and 50 inference steps on four NVIDIA H200 GPUs. The VAE is initialized from RangeLDM and the visual encoder is off-the-shelf DepthAnything3 with DINOv2 features, so the pipeline is well-specified. However, the code was not yet released at submission time, and the framework is tightly coupled to DepthAnything3 for multi-view depth and uncertainty; while the authors argue this is replaceable, they also note that independent monocular depth models would require cross-view scale alignment to avoid inconsistent geometry. Exact network dimensions beyond those in Appendix C are needed for full replication.
Generating LiDAR point clouds from multi-view camera images is a valuable task with applications in controllable data synthesis, cross-modal simulation and various perception tasks. However, most existing methods are not specifically designed for this task, and the quality of their generated LiDAR remains limited. Moreover, these methods typically focus only on semantic cues in images or generate LiDAR from a monocular camera image. In this work, we introduce LiFi (LiDAR From images), a dedicated framework for generating visually and geometrically aligned LiDAR scenes from multi-view camera images. LiFi guides the latent diffusion process through two complementary branches: geometry and semantics. The geometry branch constructs scene features from images through a reverse sampling algorithm and an uncertainty-aware geometric encoder combined with DepthAnything3, whereas the semantic branch provides high-level semantic representations via a cross-view interaction module. We further introduce a dual-branch balanced classifier-free guidance (CFG) strategy, which enhances the model's conditional generation capability while preserving the independent learning and collaborative controllability of the two branches. Extensive experiments demonstrate that LiFi outperforms state-of-the-art methods in generating high-fidelity LiDAR scenes. The code will be publicly available at: https://github.com/szzhou88/LiFi.
Pick a starting point or write your own. Challenges run in the background, so you can keep reading while the AI investigates.
No challenges yet. Disagree with the review? Ask the AI to revisit a specific claim.