VeriHarness: Scaling Agentic Verification for Long-Horizon Tasks
VeriHarness tackles the problem of verifying long-horizon agent outputs when no stronger judge model or reference answer is available at test time. The core idea is to turn the generator's own LLM into an agentic verifier by equipping it with a workspace, evidence tools, and reusable skills, then having it resolve disagreements and challenge consensus among multiple rollouts using environmental evidence rather than voting alone. This matters because repeated sampling produces complementary correct claims, but existing methods fail to identify them when consensus conceals errors or voting discards correct alternatives.
VeriHarness is a well-conceived, extensively evaluated approach to same-model verification that convincingly outperforms pool-only baselines across five demanding workspace benchmarks and two frontier models. The two-pronged design—resolving disputes over competing claims and challenging apparently unanimous consensus—is strongly motivated by the finding that '34% of consensus values are judged incorrect, while 74% of disputed claims contain a correct candidate' (Section 2, Figure 2). Evidence-backed revision lifts the average gain to roughly six points over a single rollout. Nevertheless, the method still trails the selection oracle by about 10–11 points on average, and CLI deployments show uneven transfer, suggesting the gains are sensitive to harness implementation details.
The paper's key empirical observation is its strongest foundation: on APEX-Agents, roughly a third of agreed values are wrong, and most disputed claims include a correct alternative, which directly motivates the resolver-plus-challenger architecture. The ablation in Table 4 confirms that the two components contribute complementary gains. The scale of the study is impressive: the authors release approximately 26,000 rollouts produced at over $100,000 and evaluate on five diverse benchmarks spanning documents, spreadsheets, code, and multi-file tasks, lending credibility to the headline averages.
The gap between VeriHarness and the selection oracle remains large—about 11 points on average for Gemini 3.5 Flash and 9 points for Claude Opus 4.8—indicating that the verifier frequently misses the best available rollout even when it is in the pool. The consensus challenger also exhibits a systematic blind spot: 'in about 70% of them it checked an intermediate result that was correct, while the error sat in a later step' (Appendix J), which undermines the promise of reliably catching shared errors. Furthermore, deploying the protocol inside existing agent CLIs does not consistently preserve gains; for example, VeriHarness in Claude Code on JobBench with Opus scores 42.4, close to the single-rollout baseline of 42.8 and far below the dedicated runtime's 47.6 (Table 2). The self-evolution experiments are limited to two benchmarks and one model, and they rely on access to grader feedback on a development set, which is a strong practical assumption.
The paper provides fair, controlled comparisons: every method receives exactly the same ten-rollout pool, and the 'agentic verifier (env. access)' baseline isolates the value added by the protocol and skills, recovering 'about half of the harness's gain' (Section 4.2). The aggregation baseline is weaker—it simply rewrites the pool without evidence checks—so the comparison does not rule out more sophisticated revision-based strategies. The core evidence for the consensus/disagreement statistics comes from a single benchmark (APEX-Agents) with one model (Claude Opus 4.8), which limits generalization of that specific claim, though the overall breadth across five benchmarks mitigates this concern. The paper correctly acknowledges that a stronger verifier is out of scope but notes the limitation honestly.
The authors release their code and the full rollout pool, which is excellent. The verification protocol and skill library are documented in appendices, though exact model prompts and hyperparameters such as the verifier's temperature or context limits are not reported in the main paper. The self-evolution loop requires grader feedback on a development split, so reproducing the evolved libraries demands access to the benchmark graders, which are external. The reported results use three random seeds for the verifier, but the rollout pools are frozen, which aids reproducibility. A barrier for independent reproduction is primarily cost: ten rollouts plus multi-turn verification per task is expensive, and the protocol's effectiveness depends on the human-authored skill library or a lengthy evolution process.
As LLM agents undertake increasingly complex, long-horizon tasks, verifying their outputs becomes increasingly challenging. We study how verification capability can be strengthened with a fixed base model, without access to reference answers or grading rubrics at test time. Repeated sampling yields multiple rollouts that can contain complementary correct claims, but we need a reliable verification mechanism to determine which claims to trust. We first find that disagreement often exposes correct alternatives, while consensus can conceal errors. These observations motivate VeriHarness, which turns the underlying LLM a generator uses into an agentic verifier by giving it a workspace, evidence tools, and reusable verification skills. A disagreement resolver checks competing claims against environmental evidence, while a consensus challenger tests shared claims and searches for omitted requirements. Their findings guide the selection and revision of the final artifact. Across five long-horizon workspace benchmarks and two frontier models, VeriHarness achieves the highest selection scores among the evaluated baselines. Evidence-backed revision further improves average performance, bringing gains over a single rollout to 6.2 points with Gemini 3.5 Flash and 6.4 points with Claude Opus 4.8. We further show that verification skills can self-improve from failure feedback, demonstrating VeriHarness as a novel and critical approach for scaling long-horizon agentic verification. We release the full pool of approximately 26,000 rollouts across all five benchmarks and both models, produced at a cost of over $100,000, to support future research on agentic verification.
Pick a starting point or write your own. Challenges run in the background, so you can keep reading while the AI investigates.
No challenges yet. Disagree with the review? Ask the AI to revisit a specific claim.