Select, Label, Evaluate: Active Testing in NLP

The empirical evidence for annotation cost reduction is robust: using the Inverse Probability Weighted Estimator $\widehat{M} = \frac{1}{B}\sum_{i=1}^{B} \frac{M(f(x_i), y_i)}{Nq_i}$ correctly addresses the sampling bias introduced by non-uniform acquisition functions, and the 18-dataset sweep demonstrates that AT reliably achieves sub-1% estimation error with a fraction of the budget. The adaptive stopping criterion (Algorithm 1) is well-motivated—the observation that $| \widehat{M}_{\text{random}} - \widehat{M} | < \tau$ signals convergence of the bias term to zero—avoiding the impractical requirement of fixing $B$ a priori. The multilingual extension with cost-based priors (Section 4.2) is a genuinely useful innovation for high-stakes global deployment where annotation costs vary by language.

The computational overhead of sophisticated strategies undermines the cost-saving claims. Table 2 shows Agreement takes ~5 seconds versus Random's 0.0001 seconds—five orders of magnitude slower—yet this is dismissed as "negligible" despite scaling concerns for larger embedding spaces. The Surrogate strategy is even worse (up to 59 seconds in Table 2) because it must retrain a Random Forest or SVM on accumulated labels after every selection, introducing a hidden labeling overhead that conflicts with the framing of "minimal cost." Theoretically, the stopping criterion relies on $B \to N$ for convergence (Proposition 1), which is vacuous in the regime where AT matters ($B \ll N$); the finite-sample bound in Proposition 2 shows the bias scales as $\mathcal{O}(n^{-3/2})$ only for uniform sampling, while uncertainty sampling retains an "irreducible bias independent of $n$" that is not discussed in the experimental interpretation. Finally, the Agreement strategy's reliance on an auxiliary 8-head attention layer introduces additional hyperparameters and compute not present in Random or Coverage baselines, complicating deployment.

The benchmarking scope is impressive (18 datasets, 4 embedding strategies, 3 predictors including Claude 4.5), but the main text only displays results for 6 datasets, relegating the remainder to an unverified repository. The comparison to related work is fair regarding the distinction between Active Learning (training) and Active Testing (evaluation), but the claim that this is "the first framework for AT in NLP" is difficult to verify against concurrent or recent unpublished work in the rapidly evolving evaluation-efficiency space. Notably, the Stratified Random baseline—which performs comparably to Agreement in Figure 3—requires prior knowledge of the class distribution across the entire unlabeled dataset, information that violates the stated AT premise of avoiding full-test-set assumptions; this tension is acknowledged in a footnote but not resolved in the analysis.

Reproducibility is partially hindered by missing implementation details. While the paper mentions 10 random seeds, the actual seed values are not provided. Hyperparameters for the commercial predictors (Claude Sonnet 4.5, Amazon Nova Pro) are opaque by design, though the prompts are relegated to Appendix 0.B. The Surrogate strategy's retraining procedure—critical for reproduction—lacks details on convergence criteria or stopping tolerances for the RF/SVM. No code, data preprocessing scripts, or exact dependency versions are linked in the paper, and the repository mentioned for "full results" is not cited with a persistent URL (only "our repository"). The embedding computation time (Table 1) suggests these were pre-computed and cached, but the caching strategy and storage requirements are not documented, which matters given the 77-minute DBPedia embedding time.