Finding the Right Fit: Model-Harness Interactions across Agent Tasks

The strongest empirical contribution is the cross-benchmark score matrix in Table 1, which makes it undeniable that a model's best harness depends on the task. The persistence of the openJiuwen-Kimi pairing across all three benchmarks, backed by per-task paired comparisons (Figure 2), is a genuinely useful finding. The trajectory analysis in Section 5 isolates plausible mechanisms: PI's unbounded shell turns hangs into silent failures, while openJiuwen's bounded shell returns timeouts that models can recover from. Similarly, the demonstration that final self-checks often declare success on failed tasks is a concrete criticism of current verification practices. These matched cases give the paper diagnostic depth that earn the authors' recommendations about runtime harness design.

The study reports one scored run per task with no variance estimation, so the reader cannot tell how much of a harness-model gap is reproducible versus one-off. The authors acknowledge this directly: "Model settings, tools, and execution budgets are not uniformly matched, so the observed differences do not isolate the causal effect of any single component, and run-to-run variance is not measured." Additional concerns: (1) Some harnesses were modified for the study, most notably OpenHands' context/output limits were raised to 1M/128K from a 16K default, and openJiuwen had to be assembled from its public API rather than run as an official benchmark entry point, meaning comparison is partly with custom compositions rather than stock harnesses. (2) The Kimi baseline on PI incurred 25 provider stream errors out of 62 Terminal-Bench runs, which are counted as harness failures; this inflates the apparent advantage of alternatives. (3) The 'empirical fit' definition is limited to the evaluated pairings, so claims about native pairings being non-optimal exclude many possible vendor-specific configurations.

The evidence generally supports the claims, and comparisons to prior work are accurate. The quoted 40-fold token difference from Vats and Golev matches their abstract: "Harness choice induces up to a 40× difference in tokens per solved task," and HAL's multi-scaffold, cost-aware, log-analysis approach is fairly described. The Harness-Bench citation is broadly consistent with the source paper's emphasis on execution-alignment failures, though the specific claim that "weaker models are more harness-sensitive" is not visible in the fetched abstract and should ideally be verified. The trajectory analysis is suggestive but rests on small samples: ten same-task, same-model pairs for the main coding, plus six for Kimi, before generalizing to all final attempts. This is defensible as qualitative illustration but weaker as quantitative proof of mechanism.

Reproducibility is mixed. The authors admirably release harness adapters, evaluation code, and all 6,204 scored trajectories in a common format, which is well above current practice. However, several factors would block a clean independent reproduction: only one run per task means no estimate of provider or stochastic variance; tasks with infrastructure failures were rerun and the last run counted, which is reasonable but not deterministic; OpenRouter fallbacks were disabled yet one configuration (openJiuwen–DeepSeek on ALE) had to use DeepSeek V4.1 Flash for image inputs; and harness versions used specific package versions (e.g., DSH 0.1.1-rc.2, PI 0.84.4) that may drift. Sampling parameters were left at provider defaults, so token counts and costs depend on opaque provider-side behavior that may change over time.