Articulation Upper Bounds for Language Models

The epsilon-estimation algorithm is a methodologically careful contribution: it combines unit mining, greedy subset selection with measured submodularity ratios, and capture-recapture to project an upper bound on prompt performance. The authors report that '$\epsilon$-estimation matches or beats existing prompt optimization algorithms by up to 14%, at equal budgets, across six benchmarks,' and they backtest projections with median error around $2\%$ eight units ahead. The reconstruction bottleneck is a clever unsupervised channel metric that correlates with supervised AUC at $.37$–$.68$ across domains. The most valuable empirical finding is that articulation limits are executor-relative: humans show moderate-to-high agreement on $76\%$ of the concepts that plateau for LLMs, suggesting the barrier is not universal inarticulability but model-specific tacitness.

The first worry is structural: the regime classification (rising, reaches, bounded) depends on logistic curve fits with $L \leq 1.2$, AICc model selection, and a ceiling defined by planted rules that may simply be the wrong asymptote for socially anchored concepts. Footnote 5 lists five strong assumptions behind the method — including that samplers cover the unit universe, that unit discovery probability is informative about value, and that 'the value curve saturates, which the fitted forms impose rather than test.' If any of these fail, the reported '$\epsilon$' is not a true upper bound but a sampler-relative ceiling. Second, the finding that examples barely help bounded concepts contradicts extensive in-context learning literature. The authors acknowledge this and propose that the concepts may already be internalized or at the 'edge of resolution,' but they do not reconcile this with the simultaneous claim that bounded concepts lack a stable representation in these same models. Finally, the cross-species comparison with humans is based on a literature search for $37$ of the $65$ bounded concepts rather than a controlled human study, so claims about executor-relativity are suggestive but not experimentally secured.

The evidence for epsilon-estimation beating existing optimizers is fair and budget-matched, and the reconstruction-validity correlations are statistically significant. Yet the comparison to related work on prompt optimization is operational rather than conceptual: the paper does not deeply engage with why GEPA-style reflective optimization fails to estimate its own ceiling. More importantly, the evidence for boundedness is comparative but not causal. The authors test simpler explanations in Appendix J — e.g., label rarity, collinearity, and inconsistency — and rule them out, but they do not fully rule out that a different receiver architecture or training objective could break the plateau. The conclusion calls for 'novel learning mechanisms' without specifying what they might be or how they would differ from the pretraining and post-training already tested.

The authors commit to releasing tools and data and provide extensive appendices (A through L) documenting instruments, scaling fits, codebooks, and protocols. However, several factors would block easy independent reproduction. First, the unit-mining pipeline depends on GEPA optimizer trajectories, which come from a separate not-yet-public system. Second, the 'selection panel' used for marginal screening is fixed at 100 items, but its construction is not fully detailed, and the reconstruction bottleneck requires access to the exact same 14 models (including a frontier GPT-5.6 mentioned in Appendix J.4) to replicate scaling curves. Third, the 240-item capability battery and the full prompt scaffolds are described but not inlined in the paper. Finally, at 1,270 concepts times 14 models times multiple prompt iterations, the compute budget is substantial and the exact call counts per optimization trajectory are only summarized in aggregate.