A Strong Harness Narrows the Model Gap: A Comparative Evaluation for Legislative Document-to-XML Conversion
This paper studies how to convert U.S. state legislative PDFs into exact, schema-constrained XML. The authors test the hypothesis that exact reproduction relies more on harness engineering (tools, orchestration, deterministic constraints around a model) than on model scale. They cross three Claude models with three harness tiers (zero-shot, single-agent with tools, and decomposed multi-agent pipeline) on 153 gold bills. The headline finding is striking: the weakest model in the full pipeline outperforms the strongest model zero-shot ($0.948$ vs $0.806$), and the model spread collapses from $0.68$–$0.81$ to $0.95$ under full scaffolding.
The paper succeeds at its core claim. The controlled T2-vs-T3 ablation, which holds the 47 tools constant while varying only orchestration, provides credible evidence that decomposition and persisted state matter more than model choice for this exactness task. The four-dimensional evaluation (text, structure, amendments, citations) is well-motivated for legal publishing, and the bootstrap statistics are reported transparently. The result that citation normalization—a legally load-bearing dimension—requires a deterministic lookup tool rather than a larger model is a concrete, actionable finding.
The experimental design is the strongest asset. By giving T2 and T3 identical tool access through the same MCP interface, the authors isolate orchestration as the explanatory variable. The dimension-resolved analysis is sharp: text and structure are nearly solved by any tool-using harness, while citations remain the bottleneck, cleanly localizing residual error. The paper also honestly reports complications, such as Sonnet beating Opus at T2 in citations ($p=0.004$) and the non-monotonic variance pattern where T2 is less reliable than T1.
The T2→T3 gain is attributed to orchestration, but decomposition, persisted checkpointed state, and per-agent context isolation are intentionally bundled together and never separated; the paper admits this is a limitation, but the claim reads more precisely than the evidence allows. Generalizability is narrow: only two U.S. jurisdictions, only Claude-family models, and a single proprietary schema. The citation-normalization interaction—where Haiku fails at T2 but succeeds at T3 while stronger models succeed at T2—is described but not explained; it may reflect a fundamental model capability threshold rather than a pure harness property. The absence of cost, latency, and token-usage data is acknowledged as a limitation but is material for industrial adoption.
The evidence supports the central claim that harness engineering dominates model choice for exact legal-document reproduction. The comparison to related work is fair and well-contextualized: the authors correctly note that benchmarks like LegalBench and CUAD tolerate near-misses that legal publishing cannot, and they accurately cite Zhang et al. (2026b) on the need for fixed-harness comparisons. The external sources I checked—Zhang et al. and Harness-Bench (Yao et al., 2026)—match the paper's characterization. However, the paper lacks a non-LLM deterministic baseline; it argues from a zero-shot LLM floor rather than showing the harness outperforms a competent rule-based parser, which weakens the practical claim for practitioners considering migration from legacy systems.
Reproducibility is significantly impaired. The gold set and DTD are proprietary and confidential, respectively, so independent verification of scores is impossible. The authors mitigate this by using public source PDFs and standard scorers (Levenshtein, APTED, multiset $F_1$) with fixed seeds, but the harness itself—playbooks, agent definitions, tool implementations—is not released. Each (model, harness, document) cell is a single run with provider-default temperature, so run-to-run variance is unmeasured and could be large for tool-calling agents. No token counts, latency, or per-run costs are reported, though the authors note T2 required a far higher per-run request ceiling than T3.
Legal-content reproduction requires matching the source exactly in section hierarchy, verbatim text, citations, and amendment spans; a dropped citation or mislabeled amendment carries into the products lawyers rely on. We report an industrial pipeline converting U.S. state legislation into strict, schema-constrained XML, and hypothesize that its accuracy comes from harness engineering, the deterministic tooling built around a model, more than the model itself. Our controlled ablation crosses three harnesses (zero-shot, a single tool-using agent, and a decomposed multi-agent pipeline) with three Claude models on 153 gold bills from two jurisdictions. The harness narrows the model gap: the weakest model in the full pipeline beats the strongest model alone (0.948 versus 0.806, paired bootstrap p less than 0.001). Because the two agentic harnesses share identical tools, their gap comes from orchestration: decomposition plus the persisted state and per-agent context it brings. Across four scored dimensions, the model spread at zero-shot (0.68 to 0.81) narrows to 0.95 at the full pipeline; the residual sits almost entirely in citation normalization, where the canonical identifier is retrieved from an external authority rather than generated from the weights, so a deterministic lookup tool supplies the exactness that a larger model does not.
Pick a starting point or write your own. Challenges run in the background, so you can keep reading while the AI investigates.
No challenges yet. Disagree with the review? Ask the AI to revisit a specific claim.