What the Specification Did Not Say: Specification-Oracle Gaps in Test-Graded Coding-Agent Benchmarks
Coding-agent benchmarks increasingly grade agents against upstream test suites that are paired with natural-language specifications, yet those tests often require unstated names, module paths, or literals the specification never gives. This paper formalizes that mismatch as the specification–oracle gap and implements a fast, model-free static audit for five languages, then shows that on SWE-Milestone roughly a quarter of graded milestones contain hard hidden dependencies that can block entire build units. The finding matters because it reveals that many benchmarks inadvertently measure whether agents reproduce upstream implementation details rather than whether they satisfy the stated requirements.
The paper makes a valuable and precisely scoped contribution: it defines specification–oracle gaps formally, implements a practical multilingual audit, and demonstrates that SWE-Milestone is materially affected. However, its framing that leaderboard rankings are robust to correction understates meaningful per-repository score shifts, and several causal claims about realisation remain observational rather than experimental.
The formalization is rigorous: the authors define gaps cleanly, separate under-specification (A1/A2) from chain inconsistency (B1–B3), and model blast radius in terms of build units rather than single tests. The static audit is methodologically strong—validated at 89.2% precision and 92.9% recall on held-out expert labels, running in under thirty minutes without model calls across five languages—and the release of corrected filter lists and specification addenda goes beyond mere diagnosis.
While the paper emphasizes that leaderboard rankings survive removing gap-prone tests, it simultaneously reports that go-zero’s mean score rises from 38.33 to 42.08 and its resolve rate from 4.35% to 5.71%, which undermines the framing that gaps are harmless at the aggregate level. The realisation analysis is explicitly observational, and the authors acknowledge that blocked units may differ in difficulty from comparison units, so the 1.67-fold failure lift is suggestive but not conclusive proof that gaps cause failures rather than correlate with harder tasks. Finally, the cross-benchmark prevalence comparison is explicitly exploratory with incompatible review protocols, making strong ordering claims tentative.
The prevalence evidence is well-supported by a validated static analysis, and the Go-only counterfactual—renaming a hidden identifier in the reference breaks compilation in 126 of 128 cases—convincingly isolates the mechanism. However, cross-benchmark comparisons are admitted to be procedurally incomparable, and the paper correctly notes that its behavioural gap counts are a lower bound. Related work is treated fairly, with the authors clearly distinguishing their focus on tests that reject correct behaviour from prior work on tests that accept incorrect behaviour.
Reproduction is unusually strong: all benchmark data and 35 published run logs are public, the audit code is released, and the entire analysis completes in under half an hour on standard hardware with no model calls. The main barrier is reproducing the human validation, which relied on an expert rater and author labels for precision and recall estimates. The paper also pinpoints exact generator commits and prompt templates that explain why gaps arise, though a public URL to the released audit artifact is not provided in the main text.
Coding-agent benchmarks often describe a task in natural language but grade it with tests taken from the upstream project. Passing those tests can require information the description never states: a constant's name, a module path, an exact message. We call this a specification-oracle gap: where it exists, a benchmark also measures whether an agent reproduces upstream implementation details. We formalise the gap, including the inconsistencies that chains of dependent tasks add, and build a static audit for five languages that classifies every name-, path- and literal-level dependency of every scored test (held-out precision 89.2%, recall 92.9%). On SWE-Milestone (98 graded milestones, 35 published runs), 25 milestones contain a test that depends on an unstated, non-inferable name or path. Build units with such a dependency fail in 80.6% of runs, against 48.4% for comparable units without one, and withholding a single hidden name from the reference stops its unit's tests from compiling in 126 of 128 cases. The leaderboard's ranking is robust to removing the affected tests, but they shift scores within repositories and favour agents that reproduce upstream names. Across four further benchmarks, prevalence appears to follow construction: a third or more of the tasks are affected where specifications are release notes or LLM summaries, almost none where tasks are screened or written for their tests; how much a gap costs depends on how the harness grades. We release the audit, corrected filter lists and specification addenda.
Pick a starting point or write your own. Challenges run in the background, so you can keep reading while the AI investigates.
No challenges yet. Disagree with the review? Ask the AI to revisit a specific claim.