EvoProbe: Evaluation-Driven Benchmark Evolution for LLM Agents
Static benchmarks for LLM agents saturate and face contamination, making it hard to measure true progress over time. EvoProbe tackles this by evolving cybersecurity CTF tasks in a closed loop: generate executable tasks, diagnose solver trajectories with multi-judge scoring and phase-level subtask outcomes, and feed distilled bottlenecks back into task regeneration. Its core premise is that fine-grained diagnostics—not just aggregate pass rates—should dictate what the next benchmark probes.
EvoProbe is a well-motivated framework that demonstrates a two-round benchmark evolution in cybersecurity tasks, showing a measurable drop in solver performance in the second round. The integration of verifiable subtask scoring with Borda-weighted multi-judge trajectory evaluation is methodologically coherent, and the qualitative link between round-1 failure patterns and round-2 task designs is compelling. Yet the empirical scope is narrow—only 66 tasks across two rounds, with just 18 tasks in round 2—and the design lacks an ablation isolating the effect of diagnostic feedback from examiner model quality.
The diagnostic evaluation design is the paper’s strongest aspect. It combines phase-level subtask outcome scoring with evidence-backed LLM-judge trajectories, using Borda-based weights $w_{tj}=b_{tj}/\sum_{k=1}^{N_t} b_{tk}$ grounded on each judge’s own task-solving rank. This yields actionable signals beyond binary success, and the mapping from distilled patterns to new task requirements is concrete: round-1 feedback identifying a failure to account for state resets directly produced the round-2 task Sigil Relay, which forces state setup within a single frame.
The central claim—that diagnostic feedback drives harder tasks—is undermined by confounding. The three round-2 examiners were chosen precisely because they generated the hardest round-1 tasks, so round-2 difficulty may stem from examiner capability rather than feedback. Without a no-feedback control using the same top-three examiners, the observed 11.8% relative drop in mean Outcome and 40.0% drop in mean Pass$^3$ cannot be uniquely attributed to the feedback loop. Moreover, trajectory evaluation depends on LLM judges with acknowledged residual biases, and the small sample of 18 versus 48 tasks limits statistical confidence in cross-round trends.
The evidence supports the auxiliary claim that fine-grained diagnostics reveal model differences invisible at the aggregate level: phase-level scores decline from 89.12% in Reconnaissance to 41.58% in Flag Capture, and trajectory dimensions such as Path Reasonableness score lowest on average. Comparisons to related work are conceptually fair—AutoBencher and Benchmark Self-Evolving are correctly cited as topic-driven or reconstruction-based expansions—but the paper offers no quantitative head-to-head comparison to establish that EvoProbe produces harder or more informative benchmarks than these alternatives.
The authors release source code and provide exhaustive prompts in Appendix F, including task generation, trajectory evaluation anchors, and pattern extraction instructions. However, independent reproduction faces major barriers: the validation pipeline requires agent-assisted human review to finalize tasks, the experiments depend on specific proprietary model endpoints, and the Docker environments span 66 distinct generated tasks whose exact built artifacts are not archived. Additionally, the Borda weights tied to per-task judge rankings would require replicating the exact solver pool and API versions to reproduce.
Static benchmarks for large language model (LLM) agents face saturation and data contamination risks, motivating continual updates to evaluation tasks. However, aggregate scores and success rates offer limited insight into specific problem-solving bottlenecks and provide limited guidance for subsequent task design. We propose EvoProbe, an evaluation-driven benchmark evolution framework that integrates task generation, diagnostic evaluation, and feedback-driven regeneration into a closed loop. It combines verifiable subtask outcome scoring with evidence-supported multi-judge trajectory evaluation and distills phase-level bottlenecks, success patterns, and failure patterns into diagnostic feedback for generating new executable tasks. Across two rounds of cybersecurity experiments, evaluations of eight models on 66 tasks reveal phase-level performance differences and common failure patterns beyond aggregate scores. In a cross-round comparison with the Examiner model set and solver configuration held fixed, all eight models achieve lower Outcome scores on the updated tasks, with relative decreases of 11.8\% in mean Outcome and 40.0\% in mean Pass\textasciicircum{}3. These results support using fine-grained diagnostics to guide iterative task updates and probe agents' current capabilities and limitations.
Pick a starting point or write your own. Challenges run in the background, so you can keep reading while the AI investigates.
No challenges yet. Disagree with the review? Ask the AI to revisit a specific claim.