LESS: Lightweight Evolutionary Supernet Search in Minutes
LESS introduces a lightweight neural architecture search pipeline that marries a brief fair supernet warm-up with candidate-conditioned hard-path updates inside a CMA-ES loop. The core claim is that just six balanced SGD steps per proposal are enough to sharply reduce selection regret, enabling NAS-Bench-201 searches in roughly seven minutes on a single RTX 4090 without retuning across datasets. If this speed–accuracy trade-off holds more broadly, it could lower the barrier to reproducible NAS experiments.
The paper presents a carefully controlled empirical study with a matched-pair design and explicit workload accounting. The central finding—that short, data-dependent calibration improves the architecture returned without substantially changing the best architecture encountered—is well supported on NAS-Bench-201 with 24 paired seeds and a sign-test $p=0.0066$. However, the DARTS-space transfer rests on only three searched architectures per dataset, the mass-redistribution prior is hand-tuned on the same benchmark, and cross-hardware time comparisons are appropriately caveated but still weaker than controlled benchmarks.
The strongest elements are the paired $K=0$ versus $K=6$ design, the decomposition of selection regret versus exploration quality, and the unusually thorough computational accounting. The authors explicitly distinguish the best architecture visited from the one returned, and they show that six candidate-conditioned updates shrink selection regret from $1.275$ to $0.753$ points while barely moving best-visited accuracy. The reproducibility package is also exemplary: seeds, configurations, and a behavioral reproduction gate are all documented.
The DARTS-space results are thin: only three independent searches per dataset, and all six final retrainings share a single training seed, so reported standard deviations understate true search-to-search variance. The mass-redistribution decoder prior ($r_{\text{none}}=0.55$, $r_{\text{skip}}=0.50$) is fixed after a one-time NAS-Bench-201/CIFAR-10 development run; the authors note that adapting it to other spaces remains future work. Wall-clock speedups against FairNAS and TE-NAS rely on source-reported times from different hardware, which the paper itself says do not establish hardware-normalized speedups. Finally, the non-monotonic calibration-depth curve means the frozen $K=6$ choice is not uniquely optimal, and the eight-seed ablations are too small to resolve dataset-specific tuning.
The NAS-Bench-201 evidence is strong: 24 paired seeds, a sign-test $p=0.0066$, and a matched control for candidate-independent calibration all support the claim that lightweight calibration reduces selection regret. Comparisons to FairNAS, TE-NAS, and NASWOT are presented as source-reported summaries, and the authors responsibly note that these times use different hardware and implementations. The DARTS comparison in Table 2 is less convincing because competitors use configurations developed for the DARTS space while LESS transfers frozen hyperparameters without retuning; the three-architecture sample means the error bars mainly reflect retraining noise, not search variance. TE-NAS itself reports only 0.5 GPU-hours on CIFAR-10, underscoring that wall-clock speedup ratios are sensitive to hardware and measurement conventions.
The authors release code, provide exact seeds, configurations, and a behavioral reproduction gate that confirms exact agreement of candidate records. Nevertheless, independent reproduction would still require an RTX 4090 or equivalent to match reported wall-clock times, and the DARTS retraining uses a single training seed for all three architectures, so variability is underestimated. The heavy reliance on hand-tuned decoder priors and the absence of cross-search-space validation for the mass-redistribution rule also limit transferability.
Low-cost NAS must both explore high-performing architectures and identify them reliably, yet reducing evaluation cost often weakens the fidelity of candidate comparisons. Training-free methods reduce evaluation cost by replacing learned task feedback with proxy signals measured at initialization. We introduce LESS (Lightweight Evolutionary Supernet Search), a data-driven method that combines a brief fair hard-path warm-up with discrete search under a single CMA-ES distribution. Each proposal is evaluated as its decoded hard genotype after six candidate-conditioned supernet updates. On NAS-Bench-201, LESS achieves \(93.189\pm0.467\%\) CIFAR-10 test accuracy in 409.1 seconds, coming within 0.04 percentage points of FairNAS using approximately \(1/24\) of its source-reported search time. Matched controls show that calibration improves selected validation accuracy by \(0.577\) percentage points while changing best-visited accuracy by only \(0.054\) points, indicating that its primary effect is to reduce selection regret. The frozen configuration transfers without tuning to CIFAR-100 and ImageNet16-120 with \(69.615\pm1.139\%\) and \(43.720\pm1.697\%\) accuracy. Applied without tuning to the larger DARTS space, LESS achieves \(96.95\pm0.14\%\) on CIFAR-10 and \(82.43\pm0.80\%\) on CIFAR-100, with each search completing in approximately 43.5 minutes on a single GPU. Together, these results show that short, balanced, data-dependent updates enable competitive neural architecture search across datasets and search spaces within minutes.
Pick a starting point or write your own. Challenges run in the background, so you can keep reading while the AI investigates.
No challenges yet. Disagree with the review? Ask the AI to revisit a specific claim.