From Spectral to Frobenius: When Muon Loses Its Spectral Implicit Bias
This paper studies the implicit bias of the Muon optimizer when its idealized spectral-norm steepest-descent update is replaced by the standard practical implementation: finite Newton–Schulz iterations with a positive stabilization constant. The authors prove that, contrary to the spectral max-margin behavior established for idealized Muon, the practical variant instead converges in direction to the Frobenius max-margin separator—effectively recovering the implicit bias of ordinary gradient descent. They further characterize how the iteration count, stabilization, and polynomial choice interact, and provide conditions (logarithmically growing iterations) under which spectral bias can be recovered.
This is a well-motivated and technically careful paper that closes a meaningful gap between idealized Muon theory and its widely used implementation. The core summable-perturbation reduction to gradient descent is elegant, and the systematic disentanglement of implementation choices in Theorem 6 (Table 4) is a genuine contribution. That said, the entire analysis is confined to full-batch linear classification with separable data, and some stepsize conditions and recovery guarantees are much more restrictive than what practitioners actually use.
The central result—Theorem 2—is derived cleanly by treating the stabilized Newton–Schulz update as gradient descent plus a summable perturbation (Lemmas 4 and 5), which rigorously explains why the trajectory remains at bounded distance from GD and inherits its Frobenius max-margin limit. The disentanglement in Theorem 6 is particularly strong: it shows that removing stabilization (ε=0) without increasing K does not restore spectral bias, and that for fixed K the selected direction can differ from both max-margin solutions. The recovery theorems (Theorems 7 and 8) are constructive, and the simple synthetic experiments in Figures 1 and 2 track the predicted ratios and margins accurately.
The analysis is restricted to a very stylized regime. All results assume separable linear data and full-batch updates, while Muon is popular precisely for deep neural network training where features are not linearly separable and optimization is stochastic; the paper offers no evidence that the Frobenius bias persists there. Second, the stepsize conditions in Theorem 2 are extremely conservative. For logistic loss, the requirement η ≤ 4ε/(R²u_K) is vanishingly small when ε = 10⁻⁷, and Corollary 3’s condition (Equation 10) is practically impossible to verify at runtime. Third, Theorem 8’s recovery guarantee for the quintic polynomial relies on a randomized, gradient-dependent stabilization scheme that the paper itself admits limits direct applicability to the standard implementation.
Claims are internally consistent and the cited contrasts with Fan et al. (2025) and Soudry et al. (2018) are fair. The counterexample with A = diag(2,1) is convincing and experimentally validated. However, the paper declines to directly compare its implicit-bias findings against concurrent convergence analyses of practical Muon (Kim & Oh 2026; Peng et al. 2026; Li & Tsuchiya 2026). Because those works study convergence without characterizing the limiting direction, the non-spectral directions proven here do not contradict them, but the paper misses an opportunity to clarify whether prior sufficient conditions implicitly presuppose Frobenius-like behavior or not.
The algorithms are specified in detail (Tables 1–3) and the synthetic dataset construction is described in Appendix E. Reproducing the proof of Theorem 2 is tractable given the provided lemmas, though the application to exponential loss via the smooth extension in Lemma 13 is intricate. The main obstacle to empirical reproduction is that no code release is mentioned. Additionally, while the exact quintic coefficients are stated in Equation 7, the paper does not provide closed-form values for derived quantities such as u_K and a_K at K=5, forcing independent reproduction to recompute these constants numerically or derive them by hand.
Prior work on implicit bias shows that gradient descent selects the Frobenius max-margin direction on linearly separable problems, whereas Fan et al. (2025) established spectral max-margin behavior for spectral-norm steepest descent methods, including idealized Muon with exact orthogonalization. In practice, however, Muon approximates orthogonalization using finite Newton-Schulz iterations and a positive stabilization constant. We prove that these implementation choices cause Muon to select the Frobenius rather than the spectral max-margin direction in linear classification with separable data. We also establish this behavior for both exponential moving average and Nesterov-style momentum. The key technical ingredient is a summable perturbation argument showing that the resulting parameter trajectories remain at bounded distance from those of gradient descent. We then disentangle the roles of Newton-Schulz iteration count, the stabilization constant, and the Newton-Schulz polynomial, revealing distinct asymptotic behaviors, including directions that achieve neither Frobenius nor spectral max-margin. Finally, we show that Muon can recover its spectral max-margin bias with Newton-Schulz iterations growing logarithmically over training.
Pick a starting point or write your own. Challenges run in the background, so you can keep reading while the AI investigates.
No challenges yet. Disagree with the review? Ask the AI to revisit a specific claim.