The Accuracy Well: Nonmonotone Accuracy Dynamics at Vanishing Initialization
The paper studies how classification accuracy evolves when neural networks are trained from very small initialization. It identifies the "accuracy well": a striking non-monotone trajectory where accuracy first rises rapidly, then drops significantly, and only recovers much later. The authors argue this reveals learning stages invisible to the training loss, and they support the claim with experiments across architectures and datasets, plus rigorous theory for deep linear networks.
The paper makes a fresh contribution to small-initialization dynamics by showing that classification accuracy can undergo a pronounced rise, collapse, and recovery while the training loss remains nearly flat. The empirical breadth is commendable—covering linear, ReLU, and convolutional networks on MNIST, CIFAR-10, and synthetic tasks—and the linear theory provides a rigorous backbone for the three-stage heuristic. However, the theoretical results are tightly bounded: the full three-stage dynamics are proven only for two-layer linear networks, while the deeper ($L\geq 3$) generic case leaves recovery open, and nonlinear networks lack any rigorous characterization. The extensive disclosed use of generative AI for proofs and derivations is also an unusual variable that readers must weigh.
The accuracy well is a genuinely interesting and well-documented phenomenon. The authors show it persistently across model types, datasets, losses, and optimizers, with careful controls for initialization scale, width, and depth. The linear theory is rigorous and yields sharp contrasts: direct linear optimization ($L=1$) forbids deep wells, while factorized linear networks ($L\geq 2$) can exhibit extreme accuracy oscillations, with random unary targets collapsing from perfect to zero accuracy before eventual recovery. The three-stage heuristic—target-driven organization, degeneracy bottleneck, and recruitment—provides a useful conceptual scaffold.
The main limitation is the gap between theory and experiments. The rigorous analysis is restricted to linear networks, yet the broader claims about nonlinear feature learning rest on empirical analogy rather than derivation.
For deeper linear networks ($L\geq 3$), the authors only prove organization and bottleneck under generic initialization, while recovery is left open and requires a spectrally aligned regime to establish. The margin assumptions underlying the $L=2$ theorems may also be restrictive in high-dimensional settings. Finally, the AI use statement is unusually extensive: generative AI assisted "research brainstorming, the development and refinement of theoretical ideas and hypotheses, mathematical derivations and proof development, the design and analysis of numerical experiments, interpretation of results, and literature search." While the authors state they independently checked results, the provenance of AI-assisted proofs invites extra scrutiny.
The evidence strongly supports the claims within the linear setting. The comparisons with prior work are fair and precise: the authors distinguish the accuracy well from epoch-wise double descent, example forgetting, and grokking by emphasizing that it concerns aggregate training accuracy at microscopic predictor scale. The paper's use of Jacot et al. (2021) to justify the rank-one bottleneck in deeper linear networks is accurate: Jacot et al. prove that for small initialization the gradient flow "asymptotically goes from the saddle at the origin to a rank-one saddle." However, the discussion of nonlinear networks is more a hypothesis than a demonstrated mechanism. The linear theory motivates the three-stage picture, but it does not constitute proof that ReLU and CNN accuracy wells arise from the same organization–bottleneck–recruitment process.
Reproducibility is mixed. Experimental details and additional figures are provided in Appendix C, and the figures include hyperparameter sweeps that facilitate partial reconstruction. However, the main text does not mention a public code repository, and the extensive use of AI-generated code, figures, and proofs raises questions about which artifacts were manually verified and how. The hyperparameters for MNIST and CIFAR-10 experiments appear to be described in the appendix, but independent reproduction would benefit from released code.
We study classification dynamics under small initialization and identify a pronounced non-monotone behavior of accuracy that we call the accuracy well: accuracy rises rapidly early in training, subsequently drops, and only much later recovers. We observe this phenomenon in deep linear networks, fully connected ReLU networks on MNIST, deep convolutional networks on CIFAR-10, and controlled synthetic tasks, across different losses and optimizers and in both training and test accuracy. We propose a three-stage explanation in terms of target-driven organization, a degeneracy bottleneck, and subsequent recruitment and recovery. We develop a rigorous theory of these dynamics for shallow and deep linear networks, showing in particular that high accuracy can emerge while the predictor and the change in loss remain asymptotically vanishing, before continued optimization causes accuracy to deteriorate. For a simple synthetic classification task, we prove an extreme instance in which accuracy asymptotically rises to 100%, collapses to 0, and later recovers permanently to 100%. Our results show that classification accuracy can reveal aspects of learning dynamics that are not captured by the training loss alone.
Pick a starting point or write your own. Challenges run in the background, so you can keep reading while the AI investigates.
No challenges yet. Disagree with the review? Ask the AI to revisit a specific claim.