JevCoder: When Does Decision Delegation Pay Off in Code Repair?
Coding agents consume substantial tokens as they repeatedly decide where to look, what to try, and when to stop during automated repair. This paper asks which of these decisions can be offloaded to a cheap probabilistic judge that answers fixed questions without generating explanations. The authors introduce JevCoder, a workflow built around the proprietary Jev judge to recommend initial code snippets, guide action types during repair, and trigger submission. On 400 SWE-bench tasks across three repair models, the system cuts repair-model token use by $21.6\%$–$50.6\%$ while raising observed repair rates.
The paper delivers a carefully decomposed empirical study of decision delegation in coding agents. Its strongest feature is the component-level analysis that links each judgment type to subsequent changes in repair work and token consumption, showing that delegation pays off when it changes downstream behavior rather than merely adding cheap judgments. The full-workflow evaluation is convincing within its scope, but the study is limited to a single run per task and relies on a proprietary judge whose behavior cannot be independently inspected or reproduced.
The component experiments are the core strength. The authors do not simply report end-to-end savings; they measure judgment quality at each repair stage, test interventions individually, and trace how each affects subsequent commands and tokens. The snippet-recommendation analysis is particularly nuanced: recommended snippets reduce read and search commands before the first editing command by 5.0 per task, and withdrawing them after ten requests increases mean token use by $0.538$M, demonstrating that retaining recommended context lowers overall work despite inflating context length. The comparison with same-model judging is also well-executed, showing that while same-model judging reduces repair-model tokens, adding its own judging overhead reverses the total savings for Qwen and increases runtime for DeepSeek.
The study has several reproducibility and methodological limitations. First, each task and condition is run only once, so the bootstrap intervals capture cross-task variance but not run-to-run instability. Second, the judge is a fixed version of the commercial Jev service, making independent reproduction impossible and raising questions about whether the findings transfer to other probability-based judges. Third, evaluation labels are proxies: a file is "relevant" only if the reference patch modifies it, and repair states are labeled by program rules rather than semantic correctness. Finally, patch-outcome judgment is notably weak—Jev achieves an AUROC of only $0.562$–$0.595$ on binary patch prediction versus $0.638$–$0.684$ for DeepSeek—yet the authors wisely avoid using this signal in the final workflow.
The empirical design largely supports the claims. Paired task-level comparisons with percentile bootstrap 95% intervals are used throughout, and the prespecified repair-loss criterion ($-10$ percentage points lower bound) provides a clear threshold for accepting interventions. The same-model judging baseline is a fair comparator that helps isolate workflow benefits from raw judging overhead. However, the paper does not compare against stronger contemporary baselines such as SWE-PRM or SWE-Gym verifiers in the full-workflow setting; the related work discussion positions JevCoder as orthogonal, but readers are left uncertain how the absolute repair rates compare to state-of-the-art agent workflows.
Reproduction is partially blocked by the proprietary Jev API and the anonymous artifact repository. The paper states: "We use a fixed version of the commercial Jev service and record its usage separately from repair-model tokens." Hyperparameters, prompts, and program rules are documented in the appendices, which aids replication, but the fact that each task and condition has one run means variance estimates are unavailable. Runtime comparisons are also confounded by mixed serving setups: Qwen runs locally on two NVIDIA L40S GPUs, while DeepSeek, Gemini, and Jev are remote APIs with subtracted connection-delay estimates.
A coding agent repeatedly makes decisions as it repairs code. These decisions shape repair trajectories and affect the token cost of successive model requests. We ask which repair decisions can be delegated to a low-cost probabilistic judge to reduce token use while maintaining repair quality. The judges answer questions with probabilities over predefined answers, without generating explanations. We evaluate judgment quality across repair stages and test how individual interventions change subsequent repair work. We compare these interventions with unguided repair, which uses no judge, and with the same workflow using the repair model itself as the judge. The benefit depends on the work an intervention changes, rather than the cost of the judgment alone. Based on these studies, we propose JevCoder, a workflow built around Jev, a low-cost probabilistic judge. It combines code-snippet recommendations with action-type guidance and uses judgments to determine when to submit. We evaluate three repair models, each on the same 400 tasks: 200 from SWE-bench Verified and 200 from SWE-Bench Pro Verified. Compared to unguided repair, JevCoder reduces mean repair-model token use by 21.6%-50.6%, with higher observed repair rates across all three models.
Pick a starting point or write your own. Challenges run in the background, so you can keep reading while the AI investigates.
No challenges yet. Disagree with the review? Ask the AI to revisit a specific claim.