Trainable at Depth, Solvable at Small Size: What the Loss Surface of a Polynomial Activation Dictates for Designing, Initializing and Training Networks

The deep-tier experiments convincingly establish that scale control dominates activation choice for trainability at depth. The $\ell_2$ projection rescues the polynomial network from chance to matching ReLU within $0.29$ points at depth $30$, while an unnormalized identity path takes it to $94.50\%$ at depth $100$ and $93.80\%$ at depth $300$ by allowing hidden norms to grow instead of saturating at the clamp bound. On the solvable tier, the numerical irreducible decomposition with Bertini rigorously verifies that the loss is constant on each critical component and that the number of zero Hessian eigenvalues equals the component dimension, giving strong algebraic footing to the geometric intuition. As the authors summarize in Section 6.1, "The activation choice is second-order; scale control is first-order."

A glaring internal inconsistency damages the small-model claims. The abstract states the real critical point is "the output-zero network with a vanishing Hessian," yet Section 6.3 explicitly states it has a "nonzero but indefinite (rank 2, eigenvalues $\pm 3.1600$)" Hessian and calls it a saddle. This contradiction makes the solvable-tier exposition hard to trust. More broadly, the paper admits in Section 2 that it "claims no theorem" and that claims are "restricted to measured behaviour," but later sections repeatedly assert that the small models "prescribe" or "dictate" deep-network design, a causal overreach. The evaluation is also confined to MNIST and fully-connected stacks; as noted in Section 6.5, the conclusions "do not automatically transfer to convolutional or attention architectures." Finally, the projection abruptly fails at depth $50$ for reasons the algebraic model does not explain, while the residual alternative permits unbounded growth in hidden norms (up to $15729.8$) whose practical implications are unexplored.

Within the studied regime, the evidence supports the qualitative claim that the unnormalized polynomial dies by saturation—hidden norms hit the clamp-imposed bound $226.27$—and that both the projection and the residual remedy this, consistent with the algebraic finding that their critical sets become positive-dimensional. However, the projective polynomial remains less accurate than ReLU, GELU, or SiLU at equal depth and parameter count, so the evidence does not support the polynomial as a better nonlinearity. Comparisons to gated units show collapse without normalization but do not fully tabulate their repaired performance once normalized, weakening the claim that gating confers no advantage. The correspondence between the untruncated square-loss ideal and the clamped, cross-entropy-trained deep network is asserted but not rigorously derived.

Reproducibility is strong in terms of artifacts: complete code, JSON logs, and macro-generation scripts are released under an MIT license, and the authors report that "Every number here is produced by the harness in experiments/ and injected into the text as a LaTeX macro by make_macros.py." Hyperparameters and the harness structure are documented in detail. Weaknesses include that most configurations use a single seed (42), and the collapsed training arms exhibit numerical sensitivity to thread count and accumulation order, producing spreads of several points that the authors explicitly warn should not be read to the last digit.