AdaRubric: Task-Adaptive Rubrics for LLM Agent Evaluation

The core insight—that evaluation dimensions should be a function of the task, not a fixed property of the evaluator—is convincingly demonstrated. Ablations show that adaptive rubric generation contributes +0.14 Pearson $r$ over generic fixed rubrics, which exceeds the -0.11 gap incurred when switching from GPT-4o to Llama-3.1-8B, confirming that rubric design dominates backbone capability. The DimensionAwareFilter is theoretically justified under Proposition J.2 as the minimal constraint preventing dimension-level failure masking, and empirically improves DPO success rates by +2.6 pp over AbsoluteThreshold. Cross-domain transfer experiments are particularly strong: AdaRubric preference pairs from WebArena achieve 31.2% TCR on ToolBench, outperforming Prometheus trained in-domain (29.3%), demonstrating that task-conditioned evaluation teaches generalizable quality preferences.

The theoretical analysis in Appendix J relies on a Gaussian noise model that the authors describe as a convenient idealization rather than validated against actual LLM error distributions. Rubric quality depends heavily on LLM capability and task description clarity; vague tasks yield incomplete dimensions (human study shows 4.3/5 relevance vs 4.6 for expert-designed). The SWE-bench evaluation uses only 300 sampled trajectories evaluated against a binary oracle rather than human judgments, providing weaker evidence for generalization than the main benchmarks. While the paper claims deployment-grade reliability, the computational cost remains 3–5× higher than GPT-4 Direct with 40 LLM calls per trajectory (K≈8 steps × N=5 dimensions), which may limit scalability despite caching.

The evidence robustly supports claims regarding human correlation and downstream DPO training gains through controlled comparisons against GPT-4 Direct, G-Eval, and Prometheus using identical backbones. However, the comparison to AgentHER (Ding, 2026) cites a future/under-review paper, limiting verification. The claim that AdaRubric is among the first to adaptively generate rubrics slightly understates prior work like Lu et al. (2023) which decomposed NLG evaluation into error axes, though this was for generation rather than agent tasks. The reported gains in Table 2 (+15.5 pp over base for AdaRubric-DA) conflate the contribution of the evaluation method with the training data selection; while the pairwise construction with margin-gated assurance (Equation 7) is sound, the gains depend on the specific δ_min=0.5 threshold which is not ablated.

The paper provides detailed methodological documentation including the RUBRIC_PROMPT and EVAL_PROMPT templates (Appendix A), hyperparameters ($N=5$, $λ=0.5$, $δ_{min}=0.5$), and explicit formulas for reliability computation (Equation 9). Code is promised at a GitHub repository. However, full reproduction is hindered by reliance on proprietary GPT-4o for primary results; while Table 4 provides Llama-3.1-70B ablations ($r=0.75$), the main claims use closed-model outputs. Rubric validation employs automated checks (cosine distance >0.3, weight sum to 1) but the fallback to domain-specific templates when generation fails is not fully specified. The human annotation guidelines are referenced but not provided in the available text, and the specific rubric generation retry logic (one retry then fallback) could introduce variability across runs.