Neural scaling laws and evolution of learnable activation functions of Kolmogorov-Arnold networks
This work investigates how Kolmogorov-Arnold Networks (KANs) scale with training dataset size across three recent variants—BSRBF-KAN, Gottlieb-KAN, and Faster-KAN—on MNIST, Fashion-MNIST, and a magnetic-parameter regression task. The authors fit test loss versus dataset size with a smoothly broken power law, report regime crossings from random-guess to faster- and slower-scaling branches in classification, and analyze how the shapes of learnable edge functions evolve as data grows. The central takeaway is that KAN scaling is architecture-dependent and task-dependent, and that Gottlieb-KAN edge functions collapse into a single shape family rather than diversifying symbolically.
The paper makes a useful empirical contribution by extending neural scaling analysis to KANs and documenting broken power-law behavior, but its conclusions are constrained by very short training schedules, narrow benchmarks, and the absence of a direct MLP comparison in the main text. The claim that KANs show unique RG–FS–SS scaling sequences is plausible on the reported curves, yet the evidence is insufficient to establish this as a general property of KAN architectures rather than an artifact of the specific training and model configurations tested.
The adoption of the smoothly broken power-law functional form from Caballero et al. is well-justified and the fits capture the observed curvature better than a single power law would. The activation-function analysis yields a genuinely interesting negative result: after normalization, Gottlieb-KAN edge functions collapse onto a common shape family within each layer, suggesting that much of the network's expressivity in these tasks comes from amplitude scaling rather than functional diversity.
The most serious experimental limitation is the training protocol: classification networks are trained for only 15 epochs with early-stopping patience 100, which effectively disables early stopping and likely causes undertraining in the small-data regime where the reported random-guess plateau is observed. The authors themselves acknowledge that fixed training duration is insufficient for wider regression models, stating that increasing epochs lowers losses below those of smaller models. It is therefore likely that the RG plateau and the contrast with MLPs—relegated to Supplementary Note 1—are partially artifacts of insufficient compute per sample rather than intrinsic KAN scaling behavior.
The benchmarks are also extremely narrow: two simple 10-class grayscale datasets and a single scientific regression task drawn from the authors' own prior work. Claims that edge functions evolve toward stable, interpretable symbolic forms are overstated; the paper explicitly admits that for Gottlieb-KAN, assigning distinct symbolic expressions to every edge would be impractical at scale, and the BSRBF-KAN functions vary too widely for systematic analysis.
The primary evidence supports the existence of broken scaling on the specific tasks studied, but cross-architecture comparisons are weak. The MLP baseline for classification appears only in supplementary material, and the regression MLP exponents are imported from a previous paper with a different experimental setup rather than measured under identical conditions here. The claim that KANs achieve higher exponents than MLPs therefore rests on a non-controlled comparison and should be treated with caution.
Reproducibility is hampered by the absence of released code, unspecified batch sizes, and the reliance on Google Colab without environment or runtime specifications. While the magnetic dataset is cited on Zenodo and the MNIST/Fashion-MNIST benchmarks are standard, the exact KAN architectures and training hyperparameters are largely deferred to Supplementary Notes, making independent verification difficult.
Kolmogorov-Arnold Networks (KANs) represent a compelling alternative to traditional Multi-Layer Perceptron (MLP)-based neural networks. By employing activation functions as learnable elements, KANs offer superior interpretability, making them suited for scientific domains. In this work, we investigate the neural scaling laws of KANs and the structural evolution of their learnable activation functions under dataset expansion. Specifically, we evaluate the scaling behavior of three KAN variants---BSRBF-KAN, Gottlieb-KAN, and Faster-KAN---across standard image classification benchmarks (MNIST and Fashion-MNIST) and a specialized scientific regression task (magnetic parameter estimation from domain images of moir\'e magnetic textures). Our results demonstrate that the test loss ${\cal L}$ exhibits a broken neural scaling law (BNSL) behavior as a function of the dataset size $N_D$. After passing through a random-guess regime, the loss follows architecture- and task-dependent scaling behavior. The loss crosses from a faster- to a slower-scaling branch, ${\cal L}\propto N_D^{-\alpha}$ and ${\cal L}\propto N_D^{-\beta}$ with $\alpha>\beta$ for image classification tasks. The exponents $\alpha$ and $\beta$ depend strongly on both the specific network architecture and the dataset-size regime, ranging from 0.4 to 1.5 and from 0.06 to 0.6, respectively. For the magnetic parameter-regression task, the loss follows a single scaling law with its exponent ranging from 1.28 to 2.59. Additionally, we provide a structural analysis of how activation functions refine their complexity as data volume increases, finding that dataset expansion drives a transition from simple linear-like approximations toward stable, interpretable symbolic forms. These findings provide a quantitative roadmap for the efficient application of KANs while managing the trade-off between model expressivity and computational overhead.
Pick a starting point or write your own. Challenges run in the background, so you can keep reading while the AI investigates.
No challenges yet. Disagree with the review? Ask the AI to revisit a specific claim.