OmniRay: Camera-Agnostic Monocular 3D Reconstruction with Composite Ray Fields
OmniRay tackles monocular 3D reconstruction across arbitrary central cameras without requiring intrinsics or camera-type labels at inference. Its core idea is that camera geometry has a hierarchy of complexity, so the ray field should be composed rather than represented by a single basis: a monotone radial base captures symmetric lenses, a spherical-harmonic residual handles smooth global deviations, and a free-form residual cleans up local variation, all applied as successive rotations on the sphere $S^2$. A DINOv2 backbone drives a Ray Module for $\mathbf{d}(u) \in S^2$ and a Distance Module for $\rho(u) > 0$, yielding an up-to-scale point cloud $\mathbf{P}(u) = \rho(u)\,\mathbf{d}(u)$ in one forward pass.
The paper offers a convincingly motivated and empirically strong feed-forward model for camera-agnostic monocular 3D reconstruction. An oracle analysis demonstrates that only the proposed composite representation achieves sub-$0.25^{\circ}$ fitting error across fisheye, equirectangular, and catadioptric geometries, while benchmark experiments show dramatic generalization to held-out camera types. However, the claim of being fully camera-agnostic is undercut by a clear zero-shot failure on out-of-distribution orientations, and the method slightly trails simpler baselines on standard perspective-heavy datasets.
The oracle fitting analysis in Table 1 is the paper's most persuasive result: it isolates representational capacity from network optimization, cleanly showing that a radial base alone ceilings at $22.8^{\circ}$ on equirectangular projections while the composite design breaks that barrier. The random-lens augmentation is also a genuine strength; without it, held-out catadioptric ray error collapses from $1.12^{\circ}$ to $30.87^{\circ}$, confirming that the structured residual decomposition is necessary for generalization rather than merely convenient.\n\nThese results are backed by a fair training protocol: the strongest baseline, UniK3D, is fine-tuned on the same source panoramas so that it sees identical training imagery.
The paper's 'camera-agnostic' branding overreaches. Figure 7 reveals a stark limitation: on an aerial panorama where the horizon deviates from the training distribution, the predicted rays cover only a bounded, forward-facing region rather than the full viewing sphere. This indicates that generalization depends heavily on viewpoint and orientation distributions, not just projection diversity.\n\nMoreover, on the perspective-biased CO3Dv2 dataset, OmniRay records $2.97^{\circ}$ ray error, lagging behind UniK3D ($2.68^{\circ}$) and Depth Anything 3 ($2.04^{\circ}$), suggesting that the added flexibility for exotic lenses imposes a modest penalty on conventional imagery. The held-out GoPro improvement is also marginal ($5.53^{\circ}$ versus $6.34^{\circ}$), and the staged training pipeline—first the Ray Module alone, then end-to-end with the Distance Module—adds complexity that the paper does not fully justify over a simpler joint alternative.
The evidence largely supports the core claim that a composite ray field outperforms a single-basis representation on wide-angle and omnidirectional cameras. Table 2 shows that on the held-out Nayar catadioptric camera, baselines exceed $75^{\circ}$ while OmniRay reaches $1.12^{\circ}$, and qualitative results confirm that competing methods tear or bend geometry on panoramas and fisheye images. Comparisons are reasonably fair: raw baselines are aligned by per-image median scale and, where applicable, converted to the same radial-distance gauge.\n\nThat said, the benchmark is assembled by the authors from disparate existing datasets rather than evaluated on a standard community benchmark for omnidirectional depth, and the strongest differentiators come on cameras where pinhole-biased methods are structurally disadvantaged. The paper does not evaluate on established automotive fisheye datasets such as Woodscape, leaving open how much the gains transfer to dense, real-world wide-angle imagery with complex occlusion and lighting.
Architectural and training details in Appendix A are thorough: the radial head uses $K=16$ knots, the spherical head uses $L=7$, and the two-phase training schedule specifies learning rates and DPT decoder blocks. Yet no code, model weights, or data-generation scripts are mentioned as released, which would make independent reproduction laborious.\n\nThe random-lens augmentation relies on hand-designed rejection criteria—rejecting draws within $\delta=15^{\circ}$ of the GoPro incidence-angle curve and inner/rim-angle windows for the catadioptric camera—that are sensitive to implementation. The staged training procedure, in which Ray Module components are activated in order before the Distance Module is attached, introduces additional hyperparameter and scheduling friction that would be hard to replicate exactly without the authors' codebase.
Monocular models recover 3D geometry from a single image in one pass, but most assume a pinhole camera and produce distorted geometry on the wide-angle imagery produced by fisheye and panoramic lenses. Camera-aware models remove this assumption by predicting a per-pixel ray field. However, they represent different camera geometries using a single fixed basis to capture both the overall camera projection and its finer variations. We argue that camera geometry can be viewed as having a hierarchy of complexity for representing rays. We present OmniRay, a feed-forward model whose Ray Module is composed of three components: a monotone radial base that captures radially symmetric structure, a spherical-harmonic residual for smooth deviations from it, and a free-form residual for remaining local variation. These components are composed as successive rotations on the sphere, allowing the representation to accommodate diverse camera geometries. Given a single image, OmniRay predicts a ray direction and an up-to-scale radial distance for every pixel, from which a 3D point cloud is recovered without needing camera information at inference time. Extensive experiments across different camera configurations demonstrate that OmniRay establishes a new state-of-the-art in camera-agnostic monocular 3D reconstruction.
Pick a starting point or write your own. Challenges run in the background, so you can keep reading while the AI investigates.
No challenges yet. Disagree with the review? Ask the AI to revisit a specific claim.