CAGE-NAS: Certified Functional Descent for Efficient Model Growth
Neural scaling laws favor ever-larger models, but deciding when a network actually needs more capacity remains ad hoc. CAGE-NAS proposes a theoretically grounded rule: keep the architecture fixed only while the current parameter space can certify a valid functional gradient descent step; otherwise, expand the structure in a function-preserving way. By coupling growth decisions to the geometry of the empirical tangent space, the method unifies training and architecture adaptation under a single convergence certificate. On a controlled synthetic regression task, it discovers compact three-layer MLPs that sit in the extreme top tail of an exhaustive post-hoc search, though the evaluation remains limited to a toy benchmark.
The paper offers a clean theoretical formulation for constructive NAS by leveraging adaptive functional gradient descent with certified relative-error approximations. Its strongest feature is the rigorous derivation of sufficient descent and global linear contraction under Polyak-Łojasiewicz conditions, specialized to the finite-data Hilbert space setting. However, the empirical contribution is extremely narrow: a single synthetic regression benchmark with 1024 training points, four random seeds, and three-hidden-layer MLPs under 600 parameters. The scalability claims are entirely prospective, as the dense Jacobian certification and SVD-based damping search incur costs that are prohibitive for realistic architectures.
The mathematical development is careful and self-contained. The specialization of the general adaptive FGD framework to the finite-data Hilbert space is exact, with explicit smoothness constants $L_s=2$ and PL constant $\mu=2$ for the squared-error loss. The sufficient-descent lemma and trajectory-level contraction are proven in Appendix B. The growth operations are strictly function-preserving, which cleanly separates architectural adaptation from parameter optimization and avoids loss disruptions at transition points.
The empirical evaluation is far too narrow. It covers only a single synthetic regression task with 1024 training points, four random seeds, and three-hidden-layer MLPs under a 600-parameter budget. The strong percentile rankings are computed against a post-hoc screen of 76,646 comparably tiny architectures, not against established NAS or growth baselines on standard benchmarks, and the authors acknowledge that classification and real-world tasks remain untested.
Beyond the dataset limitations, the tangent-family certification faces a structural boundary: if $p_A \geq s$ and $J_{A,\theta}$ has full row rank, the tangent space can reproduce any empirical functional gradient, so a small certificate may reflect interpolation rather than representational sufficiency. Initialization sensitivity is pronounced and the scalability discussion admits that wall-clock profiling, FLOPs, and memory costs are unmeasured future work, while the explicit dense Jacobian and SVD backend is prohibitive for realistic models.
The evidence rigorously supports the theoretical claims within the narrow experimental scope. The exhaustive post-hoc screen of 76,646 architectures and 12,240 hyperparameter pairs provides a fair internal baseline, and CAGE-NAS discovers models in the top 0.2% by test RMSE without enumeration. However, there is no direct empirical comparison to related constructive NAS methods such as GradMax, NORTH, Firefly, FRAGrow, or TINY on any shared benchmark. The retraining ablation yields mixed results: growth wins three of four seeds but loses on seed 1, and the authors note this contrasts with other growth algorithms where retraining typically improves performance. The comparison is therefore conceptually well-anchored but empirically one-dimensional.
The authors provide an official code repository and detail the synthetic benchmark construction, damping search strategy, and training algorithm in the appendices. The controlled setting computes exact Jacobians on the full 1024-sample training set, uses dense SVD, and evaluates 25 damping rungs after bisection. Reproduction is feasible for this tiny benchmark but faces serious obstacles for larger problems. The paper explicitly notes that profiling of wall-clock time, FLOPs, peak memory, and energy remains future work, and that the evaluated space is restricted to fixed-depth, three-hidden-layer MLPs with monotonic neuron additions.
The progressive growth of neural networks requires deciding when the current representation remains sufficient for optimization and when it should be expanded. CAGE-NAS formulates this decision in function space through an admissibility criterion on approximations of the functional gradient. As long as a representation enables a certified Functional Gradient Descent step, the architecture remains fixed; when the criterion fails, a function-preserving expansion is applied and the resulting representation is evaluated again. As the main instance, we study the family induced by the tangent space, using a regularized projection of the functional gradient. In a controlled setting with exact certification, CAGE-NAS produces architectures positioned above the 99.8th performance percentile by held-out RMSE among all admissible alternatives within the same parameter budget, without enumerating them during the growth trajectory.
Pick a starting point or write your own. Challenges run in the background, so you can keep reading while the AI investigates.
No challenges yet. Disagree with the review? Ask the AI to revisit a specific claim.