EvalBuilder: Can LLMs Build Efficient Benchmarks?
Long-horizon agent benchmarks like SWE-bench Verified can cost thousands of dollars and days of compute per evaluation run. This paper asks whether LLM agents can automatically construct cheaper proxy benchmarks that preserve the rankings and score distributions of expensive reference evaluations. The authors propose EvalBuilder, a meta-evaluation framework that assesses generated benchmarks along fidelity, efficiency, and robustness, with explicit tests for generalization to held-out models and freshly collected responses. The goal is to reduce the human effort and computational bottleneck of manual benchmark design.
EvalBuilder is a conceptually strong diagnostic framework, but the experimental results answer its titular question with a clear "not yet." Across two reference benchmarks and five construction agents each, overall pairwise accuracy ranges from $0.405$ to $0.689$, with several runs near random chance. Held-out generalization is especially poor—one agent drops from $0.700$ visible accuracy to $0.136$ held-out accuracy on SWE-bench Verified—indicating that the generated benchmarks overfit heavily to the candidates used during construction.
The meta-evaluation formulation is clean: an agent must reproduce the reference ordering on visible models while generalizing to held-out ones, and the three assessment dimensions—fidelity, efficiency, and robustness—cover exactly the properties a useful proxy must have. The iterative environment with a $16$-hour wall-clock budget, independent response collection, and explicit safeguards against response manipulation is methodologically thoughtful. The appendix case studies are a highlight, revealing agents autonomously searching ensemble weights and rolling back bad revisions, which adds valuable qualitative depth to the aggregate metrics.
The quantitative results are sobering. Pairwise accuracy on held-out-involved pairs is often close to chance; six of ten generated benchmarks compress score spread below $0.5$; and DeepSeek-V4-Pro on SWE-bench achieves a negative Spearman correlation despite a low MAE. Qualitative audits reveal even deeper problems: verifiers contain unimplemented branches that assign default scores to valid tasks, keyword checkers reward surface patterns over correctness, and cached judgments keyed by model name persist across fresh responses. The paper acknowledges that its safeguards "do not provide a complete audit of arbitrary verifier code," but these verifier flaws are so severe that some generated benchmarks are essentially non-functional.
The evidence adequately supports the paper's central message that benchmark construction remains hard, though baseline comparisons are missing. The authors fairly position their work against prior proxy-benchmark methods that select existing instances rather than generating new tasks, but they never compare against simple baselines such as random subsets or uniformly downsampled references. Without knowing whether a random $50$-task subset of SWE-bench would achieve similar $\mathrm{PAcc}$ and $\rho$, it is difficult to judge whether the LLM agents are adding any real value. The case-study evidence is stronger: the GPT-5.4 and DeepSeek-V4-Flash trajectories show agents fitting visible scores through superficial features and narrow task cores, which robustly supports the overfitting claims.
Reproducibility is compromised on multiple fronts. While the evaluation formulas—$\mathrm{PAcc}$, Spearman $\rho$, MAE, $\mathrm{Spread}$—are fully specified and the authors mention code availability, the construction agent prompts are described as "condensed descriptions" rather than verbatim logs, and critical hyperparameters like temperature and sampling budget allocations are omitted. Most critically, the study evaluates models that do not exist in any publicly available form (e.g., GPT-5.4, Claude-Opus-4.7, DeepSeek-V4), making independent verification impossible. The $16$-hour wall-clock budget and iteration counts are documented, but without access to the actual models or exact prompts, the reported trajectories cannot be reproduced.
Large language model (LLM) agents are increasingly used for long-horizon research and engineering, yet evaluating these capabilities remains expensive, time-consuming, and heavily dependent on human expertise. This motivates a natural question: can LLM agents themselves construct efficient benchmarks that preserve the conclusions of costly reference evaluations? We study this capability by formulating automatic benchmark development as a meta-evaluation problem, where an agent is given a task and an expensive reference benchmark and requested to construct a lower-cost proxy benchmark that faithfully distinguishes among candidate models and solutions. Based on this formulation, we introduce EvalBuilder, a framework for evaluating such benchmark-generation capabilities along three key dimensions: fidelity, efficiency, and robustness. Specifically, we assess whether generated benchmarks preserve model rankings, generalize to held-out models and freshly collected responses, and provide well-calibrated and discriminative scores while substantially reducing evaluation cost. We further provide an iterative environment in which agents can refine their benchmarks under a fixed time budget. Our framework provides a systematic testbed and benchmark for understanding when current LLM agents can develop useful evaluators and how benchmark construction itself can be increasingly automated.
Pick a starting point or write your own. Challenges run in the background, so you can keep reading while the AI investigates.
No challenges yet. Disagree with the review? Ask the AI to revisit a specific claim.