Phase Transition from Random Features to Fully Trainable Networks
This paper studies P-trainable neural networks, where a proportion P of hidden neurons are trainable and the rest stay fixed as random features, bridging random feature methods and fully trainable networks. The authors establish two-layer approximation bounds showing that a small trainable fraction can capture most of the gain when initialization mismatch is large, yielding a phase transition in rates of the form $\mathcal{O}(((P/\kappa_1^*+(1-P)/\kappa_0^*)m/\log m)^{-\alpha})$. They verify these assumptions explicitly for spherical biased-ReLU features and present numerical experiments showing analogous phase transitions in training losses across architectures with up to 15 layers.
The paper delivers a rigorous theoretical characterization of how partial trainability affects approximation for two-layer networks, establishing explicit polynomial rates and a sharp phase transition governed by initialization mismatch constants $\kappa_0^*$ and $\kappa_1^*$. The spherical biased-ReLU specialization provides concrete validation, and the numerical experiments across widths, depths, and targets offer broad qualitative support. However, general approximation guarantees for deeper networks remain open beyond a highly structured three-layer example. As stated in the abstract, "For $d=5$ and $\tau=10$, a trainable proportion of approximately $4.1\%$ captures $90\%$ of the reduction in the approximation bound from $P=0$ to $P=1$." Overall, it is a meaningful advance in understanding trainability-efficiency trade-offs, even if its depth coverage is incomplete.
The strongest contribution is the unified two-layer approximation framework. Theorem 3.6 gives the explicit rate $\sup_{\|f\|_{\mathcal{H}_\rho}\leq 1} \inf_{f_2^{(P)} \in \mathcal{F}_P(W,2)} \|f - f_2^{(P)}\|_{L_2(d\mu)}^2 \lesssim ((P/\kappa_1^* + (1-P)/\kappa_0^*) \cdot 9m/(40\log(63m/(20\delta))))^{-\alpha}$ under polynomial spectral decay and leverage compatibility. Corollary 3.10 then quantifies the phase transition: "When $\kappa_0^*/\kappa_1^* \gg 1$, a trainable proportion of order $(\kappa_0^*/\kappa_1^*)^{-1}$ suffices to capture any fixed fraction of the reduction between the endpoint bounds." The authors also verify these assumptions for spherical biased-ReLU features with explicit eigenvalue decay and mismatch constant $\kappa_0^* = 4\sqrt{2}\tau(d+1)/d$, making the abstract bounds concrete and checkable.
The core guarantees are limited to two-layer networks; the authors explicitly acknowledge that "shared upstream parameters couple downstream features through nonlinear composition in deeper networks. This complicates a direct deep-layer extension of the probabilistic kernel argument." The three-layer illustration in Section 5 is insightful but architecturally bespoke, and the evidence for deeper networks rests entirely on finite-sample training curves rather than population bounds. The theory further relies on polynomial spectral decay (Assumption 3.5) and leverage compatibility (Assumption 3.4), which restrict the target class and sampling distributions. Finally, the numerical results depart from the theory in magnitude: the caption for Figure 1 notes that "quantitative differences are expected because training also depends on finite sampling, the optimizer, and the training budget," leaving a gap between approximation upper bounds and observed optimization behavior.
The evidence broadly supports the qualitative claims but not exact quantitative predictions. The spherical biased-ReLU analysis rigorously substantiates the two-layer phase transition, and the structured three-layer example shows that depth can amplify a small trainable group via nonlinear composition. The numerical experiments across multiple targets and architectures report similar small-$P$ transitions, and the authors fairly observe that theory and training "show qualitatively similar transition shapes and sharper transitions at larger $\tau$." Comparisons to related work in Section 1.2 are accurate and well-differentiated, situating the paper between random feature models, semi-random features, and fully trainable networks without overstating overlap with pruning or lottery ticket literature.
The theoretical results are reproducible from the detailed derivations in the main text and appendices. The numerical experiments, however, lack an accompanying code release; the main text refers the reader to "experimental details in Appendix E.1" for hyperparameters and training settings. Target functions, widths, depths, and the trainable-proportion grid are specified, but exact optimizer configurations, initialization seeds, and layer-wise parameter counts are not fully tabulated in the provided text. Without public code, replicating the training curves in Figures 1 and 2 would require substantial reimplementation effort, though the spherical two-layer experiments are the most completely described.
Constructing accurate and computationally tractable approximation spaces is a fundamental task in scientific computing. Partially trainable neural networks bridge fixed random features and fully adaptive representations. This paper quantifies a phase transition pattern in approximation bounds and observes analogous behavior in training errors: a rapid but smooth initial improvement followed by diminishing returns, with a sharper transition under greater initialization mismatch. Under polynomial spectral decay with exponent $\alpha>1$ and leverage compatibility, we establish a squared $L_2$ approximation rate $\mathcal{O}((\kappa_P\log(m/\delta)/m)^\alpha)$ for two-layer networks of width $m$ and trainable proportion $P$, where $\kappa_P=(P/\kappa_1^*+(1-P)/\kappa_0^*)^{-1}$. The bound holds uniformly over the unit ball of a reproducing kernel Hilbert space (RKHS) with probability at least $1-\delta$. The constants $\kappa_0^*$ and $\kappa_1^*$ quantify the fixed and trainable groups' sampling mismatches relative to the target RKHS. For spherical biased-ReLU features in dimension $d$, we verify these assumptions with $\alpha=(d+2)/(d-1)$ and $\kappa_0^*=4\sqrt{2}\tau(d+1)/d$ for initialization scale $\tau\ge1$, and $\kappa_1^*=1$ under ridge leverage sampling. For $d=5$ and $\tau=10$, a trainable proportion of approximately $4.1\%$ captures $90\%$ of the reduction in the approximation bound from $P=0$ to $P=1$. A structured three-layer example illustrates how a few trainable first-layer neurons exploit fixed downstream blocks through nonlinear composition. Numerical experiments across diverse settings, including networks with up to 15 layers, exhibit similar phase transitions in training errors.
Pick a starting point or write your own. Challenges run in the background, so you can keep reading while the AI investigates.
No challenges yet. Disagree with the review? Ask the AI to revisit a specific claim.