ActiveSaddler: Automated Curriculum Learning for Agent Harness Optimization

The paper’s central conceptual move—separating the question of how to update a harness from which training scenarios should drive those updates—is well-motivated and clearly formalized. The empirical design is strong: ablations show that failure-pattern arms outperform both category-level and scenario-level arms, adaptive arm scoring raises failure-to-success conversion rates, and adaptive exploration yields better rollout allocation than a fixed periodic schedule. The cost analysis is particularly convincing, demonstrating that although ActiveSaddler raises optimizer-side cost per patch by roughly 45–70%, it substantially improves the end-to-end cost–accuracy trade-off by spending fewer expensive task-agent rollouts on low-value scenarios.

The method relies almost entirely on LLM calls for three critical functions—failure-pattern extraction, arm prioritization, and exploration control—yet the paper does not ablate these LLM components against smaller or cheaper models, nor does it report variance due to LLM stochasticity. This matters because the optimizer-side cost per generated patch rises to $11.44 on GAIA2 versus $7.87 for the base AutoSaddler, and it is unclear whether the gains would hold if the curriculum itself were budget-constrained. The failure-pattern extraction process, governed by prompt-based symptom extraction and normalization, lacks any formal consistency guarantee; the registry could accumulate spurious or duplicated patterns, and the paper offers no quantitative analysis of pattern quality or redundancy. Additionally, the two benchmarks differ dramatically in size (300 vs. 40 test tasks), and the large reported margin on Terminal-Bench 2.0 may be less stable given the small test count.

Second, the use of an LLM to estimate learning-progress scores $\phi_t(a) = g_{\mathrm{select}}(a, \mathcal{I}_{

The evidence credibly supports the claim that adaptive curricula improve harness optimization within the paper’s experimental setup. Comparisons are internally fair: fixed-order and difficulty-based baselines use the same optimizer, rollout budget, and model settings. The extension to GEPA shows the curriculum layer is not tied to a single optimizer architecture. That said, the non-LLM baselines—EMA persistence scoring and UCB-AIR—are intentionally simple, and the paper does not compare against stronger alternatives such as model-based RL for task selection or learned scenario embeddings, leaving the superiority of LLM-based bandit scoring less certain than claimed. The failure-to-success conversion metric is informative but correlational; it does not establish causality between any specific arm-selection decision and a final harness improvement.

The authors provide detailed appendices covering prompt templates, data splits, hyperparameters ($\tau_{\mathrm{sel}} = 0.15$), and algorithmic pseudocode, which aids replication in principle. In practice, independent reproduction will be hindered by dependence on the GitHub Copilot SDK and gpt-5.5, a potentially non-public model and proprietary toolchain at the time of writing, and by the inherent nondeterminism of LLM-based pattern extraction and scoring. The project website and code are promised but not yet available, and the paper does not specify API versions, temperature settings for the curriculum LLM calls beyond $\tau_{\mathrm{sel}}$, or caching strategies that could affect cost and repeatability.