Width and Depth Limits Commute During Training
This paper studies whether width and depth limits commute not just at initialization, but during training, for linear residual networks with $1/\sqrt{L}$ branch scaling and $\mu$P width scaling. Its core result is that under clipped gradient descent, the trained output converges in $\mathbb{L}_2$ as $\min\{n,L\}\to\infty$ for any fixed number of steps, proving the order of limits does not matter. Without clipping, commutativity holds only in probability, because the second moment can diverge with depth even after a single gradient step. The finding matters because it rigorously characterizes joint scaling limits in a tractable deep-learning model and isolates gradient clipping as a key enabler of stable infinite-depth training.
The paper delivers a clean and rigorous theoretical advance: it extends width–depth commutativity from initialization to training for linear residual networks under Depth-$\mu$P scaling. The distinction between convergence in probability under standard gradient descent and $\mathbb{L}_2$ convergence under clipped descent is sharp and well-characterized. While the restriction to linear networks is significant, the proof strategy—sequential limits with width-uniform depth discretization—is executed with care, and the explicit rates add concrete value. Overall, the work makes a solid contribution to the theory of neural network scaling limits.
The strongest aspects are the technical machinery that handles layer coupling introduced by training and the crisp negative result for unclipped descent. The authors correctly identify that gradient descent destroys the independent-increment Brownian structure used at initialization, and they build a uniform-in-width Euler–Maruyama estimate to control the depth limit regardless. The decomposition into a width-uniform depth discretization followed by an infinite-width limit of the continuous-depth system is well-motivated and carefully executed.
The analysis is limited to linear residual networks, which is a severe restriction because nonlinearities fundamentally change both forward and backward dynamics. The authors acknowledge this gap explicitly: the nonlinear experiments are purely empirical and entirely unsupported by theory. In addition, the analysis is carried out for a single training example to simplify notation, which sidesteps potential complexities of batch statistics and richer loss landscapes. The need for gradient clipping to achieve $\mathbb{L}_2$ commutativity is well demonstrated, but the paper does not explore whether alternative stabilizers could yield the same result.
The empirical section aligns well with the theory: outputs along different width–depth scaling paths converge to a common value as the joint scale grows, and the fitted width exponent matches the predicted $n^{-1/2}$ rate. The authors honestly flag that their theoretical $L^{-1/2}$ depth rate appears suboptimal, since empirics suggest a faster $L^{-1}$ decay. Comparisons to related work are precise; the paper distinguishes its $L^{-1/2}$ SDE-style regime from the $L^{-1}$ ODE-style regime studied by Chaintron et al., and accurately cites Hayou and Yang for the initialization result it extends.
Reproducibility is reasonably strong. The model, initialization, and training dynamics are specified in full, and all proofs are relegated to detailed appendices. The empirical study uses synthetic data and the public HIGGS dataset, with hyperparameters documented in Appendix G, and code is available via a linked repository. One obstacle is that the limiting trained output $f_{\infty}^{(k),\tau}$ has no closed form, so the convergence-rate experiments rely on an empirical reference constructed from the largest simulated scale, which introduces approximation误差 that is not fully quantified.
Recent work established that the infinite-width and infinite-depth limits of residual networks commute at initialization when residual branches are scaled by $1/\sqrt{depth}$. In particular, the network output converges to the same limit whether width and depth tend to infinity sequentially or jointly. We extend this commutativity result to training in linear residual networks. Under the same $1/\sqrt{depth}$ residual-branch scaling (Depth-$\mu$P) and the $\mu$P width parameterization, we prove that, for any fixed number of gradient-descent steps with gradient clipping, the network output converges in $L^2$ along every joint width--depth growth path as both width and depth tend to infinity. Consequently, the infinite-width and infinite-depth limits commute throughout any finite training horizon. Without gradient clipping, the corresponding $L^2$ convergence does not hold in general, but we establish the weaker convergence in probability.
Pick a starting point or write your own. Challenges run in the background, so you can keep reading while the AI investigates.
No challenges yet. Disagree with the review? Ask the AI to revisit a specific claim.