AtomSearch: Evidence-Grounded Tree Search for Agent Skill Optimization

The core motivation is well-articulated: local edits are easier to inspect and attribute, yet their value may only appear after later edits, so greedy hill-climbing can miss useful paths. Section 4.4 offers the most compelling evidence: in 20 of 30 completed searches (67%), the best validation state has an ancestor that was not the highest-scoring node at its depth. The reported skill sizes are dramatically smaller—SkillOpt skills are 2.3–19.7 times as long—which is a meaningful outcome for deployment and interpretability. The interface tying each edit to failed/passed executions, a hypothesis, and a scope is a clean design that aids traceability.

First, statistical rigor is lacking. The authors repeatedly label results as point estimates with no significance claims, and the test sets are small—NDR has only 100 test items, so a 6-point swing is just six examples. Table 2 reports validation-to-test gain ratios above 100% in multiple cases, meaning test improvements exceeded validation improvements; this is unusual for held-out data and suggests either high variance or that the small validation panels (18–105 items) provide unreliable signals.

Second, the ablation in Table 3 is limited to a single task–target pair (SpreadsheetBench with deepseek-v4-flash). It shows that contrastive atomic editing alone captures most of the gain (+6.78 points over base) and nearly all of the size reduction, while adding STS and outcome memory contributes only +1.43 and +1.07 additional points. This does not strongly establish that the tree-search machinery is necessary across tasks. Furthermore, several task–model pairs favor SkillOpt (e.g., SearchQA with deepseek-v4-flash and NDR with GPT-5.5), so the headline of exceeding SkillOpt on three of four tasks with deepseek-v4-flash conceals mixed results.

The comparison set is narrow: only SkillOpt is evaluated head-to-head. The related work discusses Skill-$\alpha$, SkillRevise, GEPA, and several prompt-optimization methods, but none appear in the experiments. The comparison is also apples-to-apples only under qualifications. SkillOpt reports results across three execution harnesses (direct chat, Codex, Claude Code) and calls itself best or tied on all 52 (model, benchmark, harness) cells against its own baselines; AtomSearch uses only direct chat, so the comparison omits SkillOpt's broader harness results. The authors note that SkillOpt GPT-5.5 scores are its reported default-configuration results, which may not match AtomSearch's exact experimental setup. The validation-to-test gain ratios would be more informative with repeated evaluations to quantify uncertainty, which the authors acknowledge but do not perform.

Reproducibility is hampered by multiple gaps. No code or implementation is released. The NDR benchmark is proprietary and in-house; while the authors state it is de-identified, it cannot be independently obtained. Hyperparameters for STS are listed ($c_{\mathrm{uct}}=0.05$, $B=6$, $\alpha=0.5$, rollout length 2), but the optimizer model used to generate contrastive edits and proposals is not specified by model name or temperature, and the prompt templates are not provided. Stochasticity is also unaddressed: no random seeds, no repeated runs, and no confidence intervals. Table 5 reports 16–45 optimizer calls and up to 17,200 item evaluations per run with cache hit rates varying from 15% to 69%, but the cause of that tenfold variance is unexplained. Independent reproduction would require reimplementing the full proposal pipeline and outcome memory from the prose description alone.