Lifetime-valid decisions by continually evolving AI analysts

The strongest parts are the formal characterization of failure modes and the algorithmic fix. Theorem 2 shows that individually valid analyses do not compose, Proposition 3 proves a fixed benchmark can be overfitted, and Lemma A2 shows that fixed-level repeated validation accumulates risk. Against these, continual validation uses an exact Clopper-Pearson bound with a decaying per-update risk budget $\delta_r = 6(\delta - \delta_0)/(\pi^2 r^2)$, ensuring the familywise error over an infinite horizon stays below $\delta$. The paired bound is especially elegant: "The paired bound compares the version with the reference analysis on the same simulated studies. It therefore needs to resolve only how often the version adds a rejection that the reference analysis would not make."

First, the empirical demonstration uses simplified agent configurations. Example 1 is described as "an early, simplified configuration of this broader agent," which limits confidence in how the method scales to fully open-ended LLM reasoning where allowed paths may be hard to enumerate. Second, while the authors acknowledge the gap, the guarantee does not cover off-grid scenarios: "the theoretical guarantee strictly applies to the null scenarios covered by validation. It does not automatically provide a mathematical bound for unobserved data generating patterns." The reported robustness on 666 of 690 off-grid scenarios is empirical, not proven. Third, computational cost is partially addressed but not fully quantified in the main text; the appendix notes that validation from scratch can require nearly 800,000 simulated studies per scenario for tight margins. Finally, the improvement-test baseline is allowed to reach a 0.817 Type I error rate in Example 2, which is a useful cautionary tale but arguably a strawman if interpreted as a competing safe method rather than an ablation showing why power-only checks fail.

The evidence generally supports the claims. The comparisons to related work are fair and accurately characterized. The authors correctly distinguish their goal from Sengupta (anytime-valid certificates for self-improvement) and from their own prior Reuse framework, noting that neither targets Type I error control for hypothesis testing. The Sun et al. Reuse abstract confirms it certifies "every promoted modification is a genuine population improvement on the underlying task distribution," which is a performance guarantee, not a Type I error bound. The static-benchmark failure in Example 1 and its success in Example 2 neatly illustrate that a fixed benchmark is only as good as its scenario coverage, and the criticism of Learn then Test for requiring a prespecified finite menu is accurate.

Reproduction would be difficult. Although the authors specify parameters such as $\alpha = 0.05$, $\delta = 0.005$, and $\alpha' = 0.0452$, and provide pseudocode in Algorithm 1, they do not report a public code repository, random seeds, complete prompt templates, or model snapshots. The validator must "run each version on simulated studies, either through its allowed paths or in full," and "each study is decided by one binding run, because rerunning a stochastic version until it rejects is a different procedure," but implementing this requires access to the exact bioequivalence agent and GrowthBook integration described only at high level. The reliance on specific models without pinned snapshots further complicates independent validation.