Prediction-powered Neural Architecture Search
Neural architecture search is bottlenecked by the cost of obtaining ground-truth performance labels. ppNAS addresses this by fusing a small set of labeled architectures with a large set of zero-cost proxies via prediction-powered inference. The core insight is to construct pairwise ranking supervision from both sources and debias systematic discrepancies using a rectified PPI objective, yielding state-of-the-art results under tight evaluation budgets.
ppNAS is a conceptually clean adaptation of prediction-powered inference to NAS. The paper thoroughly evaluates its proposal across multiple tabular benchmarks, zero-cost proxies, and sensitivity settings, reporting consistent gains over strong baselines. Its strongest point is the principled combination of noisy proxy rankings with scarce ground-truth labels via a rectified pairwise objective and architecture-level variance corrections. However, the scoring model is restricted to a simple linear function $g_{\theta}(X)=X^{\top}\theta$, and the paper offers neither theoretical guarantees for generalization of the learned scorer nor publicly available code, which limits the ability to verify or extend the work.
The pairwise ranking formulation elegantly sidesteps scale mismatches between ZCP scores and ground-truth accuracy by operating on ordinal preferences. The authors carefully address the statistical dependence induced by sharing architectures across pairs: "We propose to characterize variability at the level of the independently sampled architectures rather than at the level of the constructed pairs." Empirically, the method is robust across NASBench201, NASBench101, and TransNASBench, tolerates inverted proxy correlations, and requires only a single fitted model rather than an ensemble. As reported, "ppNAS performs best, reaching a 1.5 median rank versus 3 for the second-best method."
A key limitation is that the theoretical framework inherits the rectifier intuition from PPI without proving generalization for a learned ranking function over an open search space. The authors acknowledge this: "Standard PPI settings focus on estimating a target population parameter. In contrast, in NAS we want to learn a feature-dependent scoring function $g_{\theta}(X)$ that must generalize to unseen architecture specifications, where the pool of potential architectures is not fixed." A second concern is model expressiveness: the scorer is restricted to a linear model $g_{\theta}(X_i) = X_i^{\top}\theta$, which is far simpler than neural predictors used by competitors such as BANANAS, and may limit applicability to richer search spaces. Furthermore, no code or training scripts are released.
The comparisons are fair and extensive, with all methods sharing the same 30-iteration protocol. ppNAS is distinguished from warm-up approaches: ZCPBP Warmup "uses ZCP-induced rankings to warm up a predictor, but not afterwards," whereas ppNAS fuses both sources throughout fitting. The stress test is noteworthy: "Despite the ZCP misspecification, ppNAS remains competitive." This supports the claim that the rectified objective genuinely debiases noisy supervision rather than overfitting to well-specified proxies.
The paper uses public tabular benchmarks (NASBench201, NASBench101, TransNASBench), so data is available. However, the authors never mention a code repository, and several algorithmic steps---including the exact solver for the ranking objective and the Hessian pseudoinverse computation---are described at a high level. Without released artifacts, independent reproduction of the adaptive $\lambda$ tuning and covariance-based acquisition is difficult. Runtimes are reported as comparable to baselines, which is reassuring but not a substitute for open code.
Evaluating candidate architectures in neural architecture search (NAS) faces an inherent trade-off: on the one hand, reliable performance labels are limited because training and evaluating architectures is costly; on the other hand, zero-cost proxies (ZCPs) are cheap to compute at large scale but can be noisy. Yet, how to effectively combine these two sources of supervision remains unclear. In this paper, we propose PPNAS, a novel prediction-powered inference (PPI) approach for NAS. PPNAS fuses (1) a small set of architectures with observed performance labels and (2) a large set of architectures with ZCP information. To combine these two sources of supervision, PPNAS exploits the ordinal information provided by ZCPs to construct additional pairwise ranking supervision, while PPI debiases systematic discrepancies between ZCP-based and true performance rankings. We evaluate PPNAS in end-to-end predictor-based NAS, where it achieves state-of-the-art under limited evaluation budgets. To the best of our knowledge, PPNAS is the first prediction-powered approach for label-efficient NAS.
Pick a starting point or write your own. Challenges run in the background, so you can keep reading while the AI investigates.
No challenges yet. Disagree with the review? Ask the AI to revisit a specific claim.