Evidence-Gated Research: Statistically Controlled Model Adoption in Adaptive Search

The theoretical articulation of the adoption control problem is precise and well-motivated. The paper rigorously derives that under predictable routing and candidate freezing, component e-processes can be interleaved without breaking validity (Theorem 1), and that their minimum yields a valid persistent e-value (Lemma 1), which then composes with e-LOND to control persistent-target FDR (Theorem 2). The leakage diagnostic is particularly compelling: reusing decision evidence for candidate selection inflates a $q=20$ boundary-null crossing rate from $0.052\%$ to $3.802\%$, while restarting on fresh evidence restores $0.063\%$. The matched replay experiment is also well-designed because it isolates acquisition cost while holding crossing events fixed.

First, the abstract emphasizes that 'no persistent false-adoption path is observed for the audited EGR variants in that finite run,' yet the paper later notes that 'finite-run error estimates below are descriptive; formal guarantees are Theorem 2,' which risks over-interpreting a zero count in 5,000 trajectories as empirical proof of near-perfect control. Second, the persistent target requires improvement in every declared environment, a strict intersection–union criterion that the authors acknowledge brings 'possible conservatism when one environment accumulates evidence slowly,' but they do not quantify how often this conservatism eliminates practically useful challengers. Third, the experiments rely on small model banks—25 MLPs on digits and 16 MLPs on a synthetic 40,000-sample task—which are far from the large-scale AutoML or agentic discovery loops that motivate the work; the paper itself admits that 'larger prospectively sampled image, language-model, and fully agentic search loops remain important tests of external scope.'

The evidence strongly supports the formal theorems, whose proofs are complete and checkable. The comparisons to related work in Table 1 are fair: the authors correctly note that prior frameworks either address a different target (e.g., candidate risk or online e-value streams) or require extra specification to handle a changing incumbent. However, the positive empirical claims rest partly on the fact that baseline methods are tested on targets they were not designed for—development-only e-LOND is not intended to control persistent FDR—so the $0.621$ FDR for the baseline demonstrates harm but does not constitute a controlled comparison of equally optimized procedures. The matched replay over 600 frozen pairs is the strongest argument because it isolates acquisition cost while holding the decision events fixed, and the claimed $56.1\%$ evidence reduction there is well-supported.

The paper includes a detailed Appendix B with evidence units, splits, model families, search policies, budgets, Monte Carlo conventions, and a Reproducibility Statement promising runnable code, source arrays, run manifests, and SHA-256 checksums in a supplementary bundle. The betting grid, practical margins, and routing rules are explicitly listed. A limitation is that no permanent repository URL or formal data DOI appears in the main text or references, so independent reproduction hinges on the supplementary bundle being accessible and complete. Additionally, large-scale runtime overhead is explicitly not characterized: the paper states that 'runtime overhead in large training systems is also not characterized here; our experiments measure statistical evidence cost,' leaving open whether the per-candidate routing and stopping logic is feasible in expensive deep-learning training loops.