RIPPLE: Recovery Internalization through Process-Level Policy Learning and Effect-Aware Credit for Long-Horizon Agentic Coding
Long-horizon coding agents often notice failures yet keep following flawed plans. RIPPLE tackles this "Critique-to-Action Handoff Gap" by internalizing recovery supervision into a single self-correcting policy. It decomposes recovery into reflection triggering, diagnosis, and corrective action, using a prefix-only teacher and stage-specific RL credit that gates trajectory advantages with local validity signals. The result is a mean improvement of about 4.8 points across four software-engineering benchmarks without an online critic at deployment.
The paper makes a methodologically clean case that credit must be local and stage-specific to recover from failures in long coding trajectories. The intervention experiments and ablations are stronger than typical RL-for-agents papers. Yet the headline average masks a non-significant gain on SWE-bench Verified, the training pipeline hides candidate nomination behind full-trajectory hindsight, and reproducibility is hampered by undisclosed auxiliary-model details and no public artifact.
The three-stage decomposition and Eq. 11 advantage gating, $\widetilde{A}_{i,t}=h_{k}^{m}[\widehat{A}_{i}]^{+}+(1-h_{k}^{m})[\widehat{A}_{i}]^{-}$, produce measurable behavioral differences rather than just aggregate success rates. The leave-one-stage-out ablation shows corrective-action credit is the largest contributor, dropping Full-Loop Recovery from 59.6% to 43.5% when removed. Matched-prefix interventions cleanly isolate critique semantics: removing the critique lowers recovery from 62.9% to 39.8%, while a fluent wrong-direction critique is worse than removal.
The headline arithmetic mean improvement of 4.8 points aggregates benchmarks that use different metrics—Resolved@1 for the first three and a task-aligned macro average for SWE-Compass—and obscures that the Verified gain is not statistically significant. This selectivity in aggregation is concerning. In addition, the "prefix-only" teacher retrospectively mines complete trajectories to nominate candidate recovery windows before regenerating labels from prefixes, so it is not fully prospective. The limitation that "Sign gating supplies no learning signal in uniform-reward groups" means Eq. 11 is impotent whenever all trajectories in a GRPO group receive the same terminal reward, which is likely common.
The credit-assignment ladder in Table E.1 is persuasive: restricting credit to recovery events beats trajectory-wide broadcasting, and aligning it to specific stages beats event-wide generic gating. Random and shuffled sparsity controls confirm stage alignment is not just token-level regularization. However, the ICRL, MURPHY, and pivotal-aware comparisons are author-implemented "controlled mechanism adaptations" rather than reproduced baselines, making it hard to judge whether architectural differences or implementation details drive the gap.
No code, data, or model weights are mentioned as released. The training pipeline relies on four frozen auxiliary components—a generative reward model, a consistency model, snapshot test infrastructure, and an attribution judge—whose training procedures, hyperparameters, and prompts are not provided. Table C.1 lists high-level implementation settings but omits learning rate, batch size, group size $G$, KL coefficient $\beta$, and the number of training steps. Without these, independent reproduction of the RL stage is severely impeded.
Long-horizon coding agents can observe a failure and generate a plausible critique yet continue along the same incorrect trajectory. We call this the Critique-to-Action Handoff Gap. We introduce RIPPLE, a recovery-oriented training framework that internalizes process-level guidance into a single self-recovering policy. A prefix-only teacher provides evidence-grounded recovery supervision without access to future actions or terminal outcomes. After lightweight distillation, reinforcement learning assigns stage-specific credit to reflection triggering, diagnosis, and corrective action. Local validity signals gate trajectory advantages so that intermediate recovery decisions do not indiscriminately inherit terminal outcomes. Across four repository-level software-engineering benchmarks, RIPPLE improves the base policy by an arithmetic mean of 4.8 score points, with no auxiliary critic at deployment. Controlled ablations isolate the benefits of local, stage-aligned credit, and matched-prefix interventions show that critique semantics affect subsequent correction. Recovery gains concentrate where the prefix contains sufficient diagnostic evidence and the repository state remains reversible within the remaining budget.
Pick a starting point or write your own. Challenges run in the background, so you can keep reading while the AI investigates.
No challenges yet. Disagree with the review? Ask the AI to revisit a specific claim.