Groundability, Not Scale Alone: When Weak Reviewers Can Audit Strong Coding Agents
The paper asks when a nominally weaker LLM can reliably audit patches produced by a stronger coding agent. Its central claim is that groundability—the property that evidence exposes an independently checkable claim about missing behavior—matters more than reviewer scale. On a benchmark of 411 real execution-labeled traces and 101 controlled cases, the authors show that weak reviewers given official execution evidence can achieve near-perfect catch and near-zero over-rejection, while a deployable cascade of static checks and generated tests recovers only part of that upper bound. The finding that parameter count is not a monotonic predictor of review quality reframes the oversight problem from building bigger judges to building better checks.
The paper makes a crisp conceptual contribution with its notion of groundability and backs it with a carefully constructed benchmark. However, the headline result is an upper-bound diagnostic using official hidden tests that are unavailable in deployment. The deployable cascade still suffers from over-rejection of 0.66 and risk of 0.33 on held-out GPT-5.4 traces, making it a stress test rather than a practical reviewer. The honest separation of upper-bound from deployable results is commendable, but readers should not mistake the potential demonstrated in Section 5.3 for a solved deployment problem.
The benchmark construction is notably careful. The distinction between testimony, structured evidence, and grounded evidence is operationalized cleanly, and the synthetic ablation study with construction-known omissions isolates the hardest class (partial-logic) without syntactic signals. The upper-bound experiment is well-designed: evidence format is chosen on a 32-trace design set and frozen before evaluating the 122-trace held-out partition, avoiding test-set cherry-picking. The finding that "Parameter count is not a consistent predictor of quality" is well-supported across a heterogeneous ladder from 8B to 235B parameters.
The deployable cascade's over-rejection rate of 0.66 means two out of three acceptable patches are wrongly rejected. As the authors candidly note, "Generated tests are also asymmetric. A checked test that still fails on the patch has reject precision 0.82, while a passing test has accept precision 0.30." This asymmetry means the cascade can reliably flag defects but cannot safely accept. The 32-trace design set used to select evidence formats comes entirely from Django, raising questions about whether the frozen format transfers to other repositories. Most critically, the paper's strongest result relies on official execution evidence that "is an upper-bound diagnostic, not a deployable reviewer input." The gap between this upper bound and the deployable cascade is the paper's stated bottleneck, but it is also a large one.
The evidence supports the core claim that grounded evidence breaks the catch–over-rejection trade-off that plagues structured but unchecked evidence. The paper fairly distinguishes its goal—independent review of an agent patch—from prior repair systems that use generated tests to select their own patch, invoking Knight and Leveson on the independence assumption. The UTBoost finding that erroneous patches pass narrow test suites corroborates the motivation, and the SWE-bench execution protocol is described accurately. However, the cross-agent transfer claim rests on only 59 Gemini traces, and the accept/no-change sets add breadth (nine non-Python languages) but contain no defects, so they cannot validate defect-catching generalization. The comparison to LLM-as-judge biases is accurate but standard.
The paper reports temperature 0.8 and $k=5$ votes with majority voting, and explicitly states that "Design choices were made on a 32-trace set. The remaining partitions were fixed and evaluated once after those choices." This protocol is preregistered in spirit. However, no code, data, or complete hyperparameters for test generation are mentioned in the main text. One held-out trace was excluded due to an "unavailable temporary path," suggesting infrastructure brittleness. The lack of a full-set generated-test-only baseline on the test partition also limits independent comparison. The reviewer prompt template is reproduced verbatim in Appendix S3, which helps, but without runnable artifacts the exact reproduction of the cascade remains difficult.
Coding agents can return plausible patches that omit required behavior. These failures are hard to review because long traces and confident summaries often hide what was missed. We ask when a nominally weaker reviewer can reliably decide whether a patch solves its issue. We study 411 execution-labeled traces from three agents and 101 controlled cases. On 154 GPT-5.4 traces, structured but unchecked evidence raises both defect catch and over-rejection. We then provide official execution evidence as an upper-bound diagnostic. After choosing and freezing one of two formats per reviewer, five of six reviewers improve both rates on 122 held-out traces; two classify every trace correctly. Reviewer size is not a consistent predictor of quality. Because official tests are unavailable in deployment, we also evaluate a frozen cascade with patch-caused static errors and generated tests that first fail on the unpatched repository. On 121 scored held-out GPT-5.4 traces and 59 Gemini traces, its coverage is 0.89 and 0.86, risk is 0.33 and 0.26, catch is 0.76 and 0.80, and over-rejection is 0.66 and 0.67. Most false rejections occur when unresolved cases reach the reviewer. Official execution evidence shows the potential of weak review when decisive checks are available. Producing equally reliable checks without official tests remains the main bottleneck.
Pick a starting point or write your own. Challenges run in the background, so you can keep reading while the AI investigates.
No challenges yet. Disagree with the review? Ask the AI to revisit a specific claim.