C3: Rethinking Process Supervision via Confidence-Guided Credit Correction in Multi-Turn Reinforcement Learning
Multi-turn language agents trained with trajectory-level rewards frequently suffer from misguided reinforcement: tokens inside undesirable behaviors can still receive positive advantages. The paper proposes Confidence-Guided Credit Correction (C3), a plug-in operator that selectively reverses positive advantages within detector-localized undesirable spans, scaling the correction by the behavior policy's confidence. Because C3 leaves rewards, returns, and advantage estimators untouched, it is compatible with both critic-based and critic-free optimizers, offering a lightweight and general form of process supervision.
Overall, the paper isolates a genuine and well-characterized failure mode—misguided reinforcement—and proposes a targeted fix that is mathematically simple and optimizer-agnostic. The empirical gains are consistent and the comparison with turn-level process rewards is persuasive, but the reliance on hand-crafted detectors for invalid actions and query loops limits scalability, and the confidence-scaling rule, while intuitive, lacks a formal optimization justification.
The behavioral audit in Section 4.1 is compelling: the authors show that 29.07% of tokens within detector-identified undesirable spans carry positive base advantage under PPO, proving that trajectory-level success does not imply local credit alignment. The modular design is another strength; because C3 acts as a post-estimation operator on token-level advantages, it avoids redesigning reward functions or critic targets and ports cleanly across PPO and GDPO. Finally, the greedy-evaluation readout demonstrates practical impact: PPO+C3 cuts the query-loop rate to 0.064%, a 2.5× reduction over PPO and a 7.4× reduction over the process-reward baseline.
First, C3 depends on predefined, rule-based detectors for invalid actions and query loops, which narrows applicability to domains where failure modes can be explicitly enumerated; the paper does not address how to learn or generalize these detectors. Second, the correction gain $\alpha_{i}=\kappa_{i}q_{i}$ is intuitive yet heuristic, and the token-level confidence ablation underperforms span-level averaging by more than five points without a formal explanation for why aggregation is preferable. Third, the GDPO gains are modest (+0.86 absolute EM), and the PPO+C3 result on NQ is slightly below PPO, suggesting non-negligible variance across tasks.
The evidence supports the central claim that local credit correction outperforms reward-level process supervision: turn-level process rewards hurt PPO average EM (47.10% vs. 47.79%), whereas C3 improves it to 50.20%, with an especially large gain on 2WikiMultiHopQA. However, the benchmark coverage is narrow—only two environments and one model size per setting—and the paper does not empirically compare C3 against recent retrospective credit-assignment methods such as MURPHY or HiPER that target the same multi-turn agent setting, making it difficult to isolate whether the gains are unique to the correction mechanism or achievable by any additional process signal.
Reproducibility is partially strong: the authors publish exact detector rules, confidence-aggregation strategies, and matched random seeds, and state that baseline and C3 arms share identical data, prompts, turn budgets, and evaluation. That said, the training pipeline depends on LLM-generated SFT trajectories from GLM-5.2, and the code is currently hosted only on an anonymous temporary repository with a promise of full release; moreover, full-parameter RL on Qwen3-8B/4B is computationally expensive, creating a high barrier to independent verification.
Large language models are increasingly trained as multi-turn agents that interact with external tools. However, outcome-only reinforcement learning can produce a form of misguided reinforcement: even on successful trajectories, undesirable local behaviors may receive positive token-level advantages and be favored by the policy update. To address this issue, we propose Confidence-Guided Credit Correction (C$^3$), a plug-in method for local credit correction. C$^3$ identifies spans corresponding to undesirable behaviors and selectively corrects positive credit after advantage estimation, with the correction strength adaptively controlled by the behavior policy's confidence in the corresponding behavior, while leaving the task reward and original credit estimation unchanged. Across multiple multi-turn benchmarks, C$^3$ improves performance under both {critic-based and critic-free reinforcement learning methods}. Injecting the same behavioral signals as turn-level or outcome rewards does not consistently improve performance. Our results show that local credit correction provides an effective form of process supervision for outcome-driven RL.
Pick a starting point or write your own. Challenges run in the background, so you can keep reading while the AI investigates.
No challenges yet. Disagree with the review? Ask the AI to revisit a specific claim.