MECHVAR: Variance-Guided Mechanism Discrimination for Autonomous Machine Learning Experiment Selection
When a machine-learning benchmark improves, multiple mechanisms could explain why. This paper studies how an autonomous agent should select diagnostic probes from a finite library to discriminate among competing explanations under a tight budget. MECHVAR scores each probe by the posterior-weighted variance of its predicted responses, showing that under a shared Gaussian predictive scale this is exactly proportional to the classical Box–Hill pairwise-KL criterion. The reformulation yields $O(KE)$ vectorized rescoring, an explicit pairwise audit trail, and orders-of-magnitude faster acquisition than quadrature-based expected information gain.
MECHVAR is a narrow, well-scoped contribution: it reformulates a classical model-discrimination objective so that scoring a finite probe library costs $O(KE)$ instead of expensive mixture integrals. The experiments show it handily beats confirmation-first and score-greedy policies in moderate misspecification regimes, but they fail to statistically separate it from quadrature EIG, suggesting the practical benefit is primarily computational and diagnostic rather than statistical. The paper is honest about these boundaries and usefully distinguishes mechanism discrimination from benchmark-score maximization.
The shared-Gaussian equivalence is exact and cleanly proven: Proposition 1 shows that the posterior-weighted variance $\mathrm{MDV}_t(e)$ equals $\sigma_b^2$ times the Box–Hill pairwise-KL sum, preserving the argmax while exposing an additive pairwise audit. The speed benchmark is also striking, and the held-out Digits loop usefully demonstrates that optimizing benchmark accuracy (score-greedy) is not the same as identifying the mechanism.
The empirical head-to-head against EIG is largely unresolved: in the 25-block stress audit the Holm-adjusted $p$-values for MECHVAR versus EIG are uniformly $1.0$ across misspecification levels, and the held-out Digits AUC difference is insignificant ($p=0.0701$). The shared-scale assumption is a hard boundary: with log-scale dispersion $\delta=0.50$, MECHVAR agrees with the correct heteroscedastic Box–Hill ranking on only $25.7\%$ of states. Moreover, the method requires a pre-enumerated mechanism set and response templates; it cannot generate hypotheses, handle multivariate responses natively, or recover when the true mechanism is outside the library.
Comparisons are fair and internally consistent: EIG, confirmation-first, score-greedy, and random all operate on the same posterior and Gaussian belief model, and the six-node quadrature is validated against 50-node reference (99.7% top-probe agreement). The distinction between mechanism-discrimination and benchmark-score maximization is demonstrated cleanly on Digits, where score-greedy reaches only $0.7825$ normalized AUC versus $0.8975$ for MECHVAR. However, the bulk of the evidence is synthetic or uses planted shortcuts; the single real-data loop is small-scale scikit-learn Digits with only six mechanisms and seven probes.
The experimental protocol is documented with unusual care: paired Wilcoxon tests with Holm adjustment, explicit block randomization, disjoint calibration and held-out seeds, and exact software versions (Python 3.13.5, NumPy 2.3.5) on a specified CPU. Yet no open-source code, data repository, or hyperparameter configuration files are referenced in the manuscript; the Reproducibility Statement only guarantees the arXiv source archive contains materials needed to compile the paper, not to rerun experiments.
Benchmark gains are often mechanism-ambiguous: reproducing an improvement does not by itself identify why it occurs. We study finite-library mechanism discrimination, where posterior-weighted candidate mechanisms, executable probes, and a limited experimental budget define a sequential experiment-selection problem. MECHVAR selects the next probe by maximizing the posterior-weighted variance of its predicted responses. Under a shared-Gaussian predictive model, this score is exactly proportional to the classical Box--Hill posterior-weighted pairwise-KL criterion, yet it admits O(KE) vectorized rescoring and a transparent additive audit over mechanism pairs. A local expansion further links the score to expected information gain (EIG) when predicted response separations are small. In a 25-block stress audit, MECHVAR outperforms confirmation-first in several moderate misspecification regimes, while its primary comparisons with EIG remain statistically unresolved. In a held-out Digits loop, normalized mechanism-identification AUC is 0.8975 for MECHVAR, 0.7825 for a score-greedy policy, and 0.9092 for EIG. At K = 100, E = 200, median single-thread full-library scoring is 10.36 microseconds for MECHVAR versus 57.69 ms for six-node quadrature EIG in the recorded environment. MECHVAR therefore provides a lightweight, auditable acquisition rule for finite-library experiment selection when a shared predictive scale is a defensible approximation.
Pick a starting point or write your own. Challenges run in the background, so you can keep reading while the AI investigates.
No challenges yet. Disagree with the review? Ask the AI to revisit a specific claim.