Continuous-depth Deep Gaussian Processes
Standard deep Gaussian processes become increasingly ill-conditioned as depth grows because discrete layer composition amplifies parameter perturbations. This paper proposes replacing the discrete stack with a latent continuous-depth SDE whose drift family carries a GP-based prior, yielding two models: CDGP with a direct separable GP prior, and FlowDGP with a flow-evolved base GP. The central claim is that depth in probabilistic regression is better modeled as continuous latent evolution than as a discrete composition, and that retaining a GP prior over the drift improves stability and competitiveness relative to both discrete DGPs and neural continuous-depth alternatives.
The paper presents a well-motivated redesign of deep GP regression. Its strongest feature is the sensitivity analysis that links discrete composition pathology to a continuous-depth alternative, and the shared variational framework elegantly accommodates two cost-quality trade-offs. However, the deepest theoretical guarantee—uniform sensitivity bounds—requires a contractivity assumption that the authors explicitly note is not enforced by either CDGP or FlowDGP, leaving a gap between motivation and training dynamics. Empirically, FlowDGP is competitive across scales but does not uniformly dominate neural SDE or transformed GP baselines. The work is a useful contribution with a crisp narrative, though some claims marginally outpace the evidence.
The motivation is sharply articulated: Proposition 3.1 shows that the Lipschitz constant of the single-sample discrete DGP objective can grow exponentially with depth because of repeated layerwise Jacobian products. The continuous-depth formulation replaces this with a pathwise SDE sensitivity bound in Theorem 4.1, and the variational framework unifies CDGP and FlowDGP through a shared simulator and sparse inducing-point objective. This structural clarity is valuable. The two-model design is also practically useful: FlowDGP trains roughly $5$–$14\times$ faster than CDGP while maintaining comparable or better metrics in many regimes.
A central issue is that the contractive sensitivity bound in Corollary 4.2 is not realized by the proposed models. The authors note that "neither CDGP nor FlowDGP explicitly enforces that regime," and call the corollary an intentionally "specialized auxiliary result, not a property used to design our models." This leaves the theoretical motivation as interpretive rather than a concrete training guarantee.
Structurally, CDGP's separable kernel cannot capture joint state–depth interactions, and FlowDGP's depth evolution is deterministic given the base GP $f_0$, so drift uncertainty across depth is limited. These limitations are honestly acknowledged in the Limitations section but do restrict generality. Empirically, the benchmark comparison is mixed: TGP wins on two NLL comparisons and neural SDE wins on nonstationary RMSE and uci_energy NLL, so the claim of a "stronger overall story" is slightly selective.
The evidence supports the claim that continuous depth improves stability over plain DGPs, and the toy study clearly visualizes the discrete-depth failure mode. Residual DGP is correctly positioned as a meaningful but incomplete repair. However, the comparison to related work is nuanced: while FlowDGP outperforms neural ODEs on probabilistic metrics, neural SDE is highly competitive, achieving "very strong RMSE on nonstationary and sarcos_small." At large scale, only FlowDGP is tested against neural alternatives because CDGP and Residual DGP are omitted for cost reasons; this means the scalability argument rests on a single model rather than the full family. The comparisons are otherwise fair, with direct baselines for both discrete and continuous depth included.
The experimental protocol is unusually well documented: dataset descriptions are explicit, hyperparameter grids are reported in Appendix E.2, and the exact $S=300$ Monte Carlo predictive procedure is stated. Yet no code repository URL is provided, which significantly blocks independent reproduction given the complexity of the continuous-depth sparse variational simulator and the FlowDGP ODE solver integration. The reliance on a specific left-point Euler–Maruyama discretization with fixed depth grid is clear, but verifying the implementation of drift transport and the Kronecker-structured inducing-point covariance for CDGP would require substantial reimplementation effort without released code.
Deep Gaussian processes (DGPs) are appealing Bayesian regression models, but their standard discrete-depth parameterization often becomes harder to optimize as depth grows. We argue that depth is better treated as a latent continuous evolution than as a long discrete stack. We therefore propose a continuous-depth formulation in which the latent representation follows a stochastic differential equation with a depth-indexed drift family equipped with a GP-based prior. This reframes the core design choice as the prior placed on that family, rather than adopting a purely neural continuous-depth parameterization as in neural ODE or neural SDE models. We study two instantiations of this idea: CDGP, a direct GP prior over the state--depth drift, and FlowDGP, a flow-evolved prior that captures richer depth dependence at lower training cost. To motivate the redesign, we analyze how the single-sample DGP objective can become increasingly sensitive under repeated layer composition, while residual DGP is a useful first repair that still remains a finite discrete construction. We also give pathwise sensitivity analysis for the continuous-depth formulation. Experiments on synthetic, benchmark, and large-scale regression tasks show that residual DGP improves over plain DGP, while CDGP and FlowDGP provide a stronger overall story across the regimes we study.
Pick a starting point or write your own. Challenges run in the background, so you can keep reading while the AI investigates.
No challenges yet. Disagree with the review? Ask the AI to revisit a specific claim.