QuaNAS: Uncertainty-Aware Reuse of Earlier Searches for Query-Efficient Neural Architecture Search
QuaNAS tackles query-efficient neural architecture search by treating a library of earlier searches as explicit but uncertain knowledge. Its core idea is to encode design decisions as additive main-effect priors that seed a Gaussian process, letting every new evaluation re-weight trust in transferred knowledge and fall back on zero-cost proxies when the library is uninformative. The paper tests this under a frozen, self-preregistered protocol on 12 held-out datasets spanning images, time series, text, and tabular data.
The paper makes a strong, well-supported case that reusing earlier searches via inspectable main-effect priors yields large efficiency gains at small budgets. The frozen protocol, extensive ablations, and honest reporting of mixed results lend credibility, but the evidence for the necessity of online re-weighting is weak on the held-out data, and the re-implementation of RoBoT raises questions about whether the reported gaps are fully attributable to QuaNAS.
The central empirical result—that the architecture prior from earlier searches cuts anytime regret by roughly $3\times$ relative to scratch baselines and that the same prior alone finds competitive architectures without any query—holds up across modalities and is robust to library manipulation. The experimental hygiene is exemplary: a frozen protocol, a test-split firewall, deterministic search with committed random seeds, and faithful failure reporting.
Three issues deserve scrutiny. First, the ablation removing online re-weighting by fixing the prior weights at their meta-learned values ($\sigma_a=\sigma_s=0$) actually yielded a slightly lower mean regret on the 12 confirmation datasets (0.056 vs 0.058); the authors defend this component as a safeguard against rare failures seen during development, but the held-out evidence is thin. Second, the primary comparator RoBoT is an in-house re-implementation with a different kernel and acquisition function, which complicates attribution of the gap. Third, the reference models are intentionally simple (ResNet-18, fastText, CatBoost), so the claim that final models match or exceed them speaks to matched-recipe utility rather than state-of-the-art performance.
The comparisons with post-hoc transfer methods are nuanced and informative: QuaNAS ties RGPE in mean regret but wins on tabular tasks with a lighter tail, and the authors correctly identify that per-task surrogates dominate when earlier tasks share exact candidates. The zero-cost proxy claim is well-cited and consistent with internal data, though proxies still serve a secondary screening role inside QuaNAS.
Reproducibility is generally strong: every hyperparameter is fixed and reported, random seeds are listed, and the authors report that rerunning released code reproduced AUCs to within $10^{-14}$. The main barrier to independent reproduction is the construction of the 47 pool library; while the authors commit to releasing all tables and code, the complexity of modality detection, quality-control fallbacks, and library assembly means that exact reproduction without their artifacts would be time-consuming.
Neural architecture search (NAS) on a new dataset often affords only tens of short training runs, so what the search knows before its first evaluation decides its efficiency. Yet most NAS methods start from scratch, guided at best by zero-cost proxies that are unreliable across tasks. QuaNAS treats a library of earlier searches in the same search space as explicit knowledge of uncertain reliability. Main effects of design decisions learned from this library and, secondarily, zero-cost proxies become priors over candidate architectures without any label of the new dataset. A Gaussian process starts at the prior optimum and re-estimates after every evaluation how far each prior is trusted. Under a protocol frozen before any confirmation data were seen, QuaNAS was evaluated once on 12 new image, time-series, text and tabular datasets. It lowered anytime regret on the short-training objective 2.7- to 3.9-fold relative to five pre-specified NAS baselines that start from scratch (Holm-adjusted p = 0.0012 each). Its final models matched (non-inferiority margin 0.5 pp) or exceeded matched-recipe reference models on nine datasets, as did the prior optimum, selected without a single evaluation. Post-hoc comparisons with established transfer methods given the same library trace the gain to the library: without it, the same Gaussian process had over twice the regret. They also show where each transfer mechanism works best: the pooled main effects of QuaNAS for weakly transferable tabular tasks, per-task surrogates for image, time-series and text tasks and for small or mismatched libraries. Finally, the main effects make the transferred knowledge inspectable.
Pick a starting point or write your own. Challenges run in the background, so you can keep reading while the AI investigates.
No challenges yet. Disagree with the review? Ask the AI to revisit a specific claim.