Removing spurious minima for planar features by skip connections
Understanding when neural-network loss landscapes contain spurious local minima is a foundational problem in deep learning. This paper studies shallow bias-free ReLU networks in a Gaussian teacher-student setting and proves that a learned linear skip connection eliminates all spurious local minima for planar teacher features with positive output weights once the student is at least as wide as the teacher. In contrast, without the skip, spurious minima can persist at every width—a striking separation between architectural bias and overparameterization.
The paper delivers a rigorous and significant theoretical result: for teacher networks whose feature directions lie in a 2D subspace and have positive output weights, adding a learned linear skip guarantees that every local minimum with non-negative student output weights is an exact fit whenever $n \geq m$ (Theorem 10). This is complemented by a tight negative result showing that plain ReLU networks can harbor spurious local minima even under arbitrary overparameterization (Theorem 11). The analysis is mathematically deep and supported by Lean certificates, though the planar and non-negative-weight restrictions limit the scope, and the results do not imply convergence guarantees for gradient descent.
The loss decomposition $\mathscr{L}_{\mathrm{skip}}(b^{\mathbf{s}},s,w) = \mathscr{L}_{C}(s,w) + \frac{1}{2}\|b^{\mathbf{s}} - b_{\rm match}(s,w)\|_{2}^{2}$ (Lemma 5) is a clean and powerful tool that reduces skip-network analysis to a centered absolute-value network plus a quadratic term in the skip. The construction of persistent spurious minima for plain networks (Theorem 11) is explicit—three teacher neurons at angles $(0, 5\pi/6, 4\pi/3)$—and the proof combines a strong-convexity box certificate with a conceptual analysis of kinks. Additional structural results, such as feature confinement to the teacher span (Proposition 8) and the sharp effective-width bound (Proposition 12), are strong and correct.
The central limitation is the planar-feature assumption: the teacher directions must lie in a 2D subspace. The authors conjecture this can be removed, but the entire interlacing and descent analysis relies on the circular ordering of directions and does not generalize to higher-dimensional teacher subspaces (Section 7). The non-negative student output weight restriction is also strong and excludes many natural parameter configurations. Furthermore, the empirical-loss transfer (Theorem 13) only applies to $r$-stable minima—those persistent on fixed-radius parameter balls—so it does not rule out ordinary spurious empirical minima. Finally, as a pure landscape theory, the results do not imply that any practical optimization algorithm will find the global minimum.
The comparison to Zhou et al. (2021) is accurate: their work requires sufficiently separated teacher features and a low-loss initialization for convergence, whereas this paper characterizes the global landscape for planar teachers without separation. Fetching the cited source confirms Zhou et al. frame their result as requiring that "the loss is already lower than a threshold (polynomial in relevant parameters)". The effective-width bound is sharp, with an explicit construction attaining $2m$ distinct student directions for odd $m \geq 3$. The negative result for plain networks is concrete and withstands scrutiny, though it is specific to input dimension two.
The authors provide Lean 4 certificates in a linked repository, which is an excellent step toward formal verification and reproducibility of the mathematical claims. However, there are no numerical experiments, no released code for the box-certificate computation behind Theorem 11, and no hyperparameters to tune because the paper is purely theoretical. Independent reproduction is therefore possible in principle via the theorem prover, but a reader wishing to verify the computer-assisted construction numerically would need to reimplement the certificate from scratch. The first nine pages summarize results and proof ideas, while the full formalization resides in appendices and Lean files.
Understanding loss landscapes is central to explaining neural-network training, yet their structure remains only partially understood even in simple models. We study the Gaussian population loss of shallow, bias-free ReLU networks in the teacher--student setting. This provides a simple model for studying essential aspects such as feature learning and overparameterization. For teacher networks with positive output weights and planar features, we show that including a learned linear skip removes all spurious local minima with non-negative student output weights once the student network is at least as wide as the teacher network. In contrast, without the skip, we construct a fixed teacher network with positive output weights and only three hidden neurons in input dimension two whose spurious local minima persist at every student width at least three. Thus, a learned linear skip can remove spurious minima that persist under arbitrary overparameterization. Furthermore, we show that a positive output weight student network always learns the subspace spanned by the teacher features: student features at local minima with non-negative student output weights lie in the span of the teacher features. For ReLU networks in two dimensions, even heavily overparameterized student networks have effective width controlled by the teacher width: every critical point with positive student output weights has at most twice as many distinct student feature directions as teacher neurons. Finally, we transfer the benignity result to empirical minima over parameter balls of any prescribed radius, with the required sampling accuracy depending on that radius.
Pick a starting point or write your own. Challenges run in the background, so you can keep reading while the AI investigates.
No challenges yet. Disagree with the review? Ask the AI to revisit a specific claim.