Doc2Harness: Benchmarking LLMs on Long-Horizon Harness Engineering of Coding Agents
Doc2Harness evaluates whether frontier LLMs can engineer a coding-agent harness from scratch, a capability that existing SWE benchmarks ignore by holding the harness fixed. The benchmark asks a build model to generate a complete harness from a development document and an interface-only skeleton, then evaluates it deterministically with a scripted mock model across $328$ black-box test cases. The results reveal a $26$-point performance gap between the best and worst models, with even the strongest scoring only $53.05\%$, suggesting substantial room for improvement in long-horizon infrastructure construction.
Doc2Harness is a methodologically careful first step toward measuring harness engineering, with strong experimental controls and a discriminative test suite. The two-stage design successfully isolates harness correctness from driver instability, yielding a reliable spread of scores. Nevertheless, the single-build protocol and the large number of universally failed floor cases mean the benchmark likely underestimates capabilities and may contain unreachable requirements, tempering the strength of its conclusions.
The benchmark's strongest features are its deterministic mock-model evaluation and high internal consistency. By driving the candidate harness with a preset script instead of a live LLM, the authors eliminate driver variance and ensure that PASS/FAIL reflects harness behavior alone. The suite is also a reliable instrument: the KR-20 reliability coefficient of $0.913$ indicates the test cases tap a single underlying capability, and the $26$-point gap between GPT-6 Astra ($53.05\%$) and Hy3 ($27.44\%$) confirms the suite is neither saturated nor unable to discriminate.
The protocol deliberately prevents models from receiving behavioral feedback during construction, which makes the task artificially difficult and less ecologically valid. As the authors acknowledge, 'the contract check tests form rather than behavior, so the amount of revision a candidate harness receives depends on when the model submits, and the pass rate of an early submitter is a lower bound rather than an upper bound' (Section 6). In practice, engineers test and debug their harnesses iteratively, so a single-shot, form-only check likely suppresses scores.\n\nA second issue is test-suite composition. Out of $328$ cases, $56$ are floor cases failed by every model, providing no discriminative signal and suggesting that roughly $17\%$ of the specification may be unrealistic or ambiguous for the current cohort. Moreover, the single-variable expansion method generates correlated variants: 'a single defect can therefore cause correlated failures across these variants' (Section 6), meaning the total failure count overstates the number of independent bugs.
The comparisons to prior work are fair and well-anchored. The authors correctly note that 'existing SWE benchmarks such as SWE-bench Pro, NL2Repo-Bench, and Commit0 judge whether the model's output is acceptable using the interactive unit tests supplied with the task. Such criteria do not transfer directly to harnesses' (Section 1), because harnesses can implement the same feature in many ways and must be driven to be judged. The architectural analysis is less convincing: the reported partial correlation of $-0.423$ between module count and pass rate (Section 4.3) is merely observational, and the causal claim that 'fragmentation is therefore not free' would require controlled experiments, not just a regression on model-generated artifacts.
Experimental protocols are documented with unusual precision. The builder harness is pinned to OpenCode $1.17.0$, builds are limited to $24$ turns or a two-hour wall clock, and each test case runs in its own container with a $900$-second timeout (Section 3). What is missing is an explicit artifact release plan. Although the author list includes a project URL (geyuyao.com/doc2harness), the paper does not state whether the mock-model scripts, test fixtures, or positive-control implementations will be released. Without these exact artifacts, independent reproduction of the $328$-test-case suite is blocked.
Agents driven by large language models are increasingly deployed on long-horizon tasks, where they use tools and modify real workspaces. How well they perform depends as much on the harness around the model as on the model itself. Existing benchmarks measure downstream task success under a fixed harness, leaving harness engineering itself unexplored. We propose Doc2Harness, a two-stage benchmark that evaluates long-horizon harness engineering directly. In Stage 1, a build model operating inside a fixed builder harness receives an interface-only contract skeleton and a development document that consolidates behavior across twelve frontier harnesses, and then constructs a complete candidate harness in an isolated container. In Stage 2, the frozen candidate harness runs in containers separate from the construction environment, one per test case, and is driven end to end by a scripted mock model. We score each test case using four observable channels, namely the requests sent to the model, the files on disk, the process exit code, and the command output; positive controls rule out vacuous passes. Across nineteen frontier models, each evaluated with three independent builds on 328 test cases, GPT-6 Astra achieves the highest pass rate at 53.05%, while the weakest model reaches 27.44%, a gap of roughly twenty-six percentage points. Even the strongest model fails nearly half of the test cases, and failures cluster around how cross-cutting functionality is split across modules. To our knowledge, Doc2Harness provides the first deterministic benchmark for measuring long-horizon harness engineering.
Pick a starting point or write your own. Challenges run in the background, so you can keep reading while the AI investigates.
No challenges yet. Disagree with the review? Ask the AI to revisit a specific claim.