Beyond Corrected Actions: Training LLM Agents on Textual Critiques
LLM agents frequently fail in multi-turn settings by violating policies, using tools incorrectly, or acting prematurely. This paper proposes Critique-Augmented Agentic Fine-Tuning (CAFT), which rolls out a student agent, has an expert label each wrong turn with both a textual critique and a corrected action, and then fine-tunes the student to generate that critique and correction while keeping the original mistake in context. The central finding is that these natural-language explanations provide learning signal beyond action-only supervision, improving task success across four models and two benchmarks.
CAFT offers a conceptually clean and empirically supported method for improving multi-turn LLM agents by exploiting the expert's ability to articulate why a mistake occurred. The central ablation—showing that training on critiques alone consistently outperforms training on corrected actions alone—is strong evidence for the paper's core thesis. However, the broader claims about reinforcement-learning warm starts and iterative training rest on thin evidence, and the evaluation domains remain narrow.
The paper's main claim is bolstered by careful experimental design. By controlling the trajectory source and expert model while varying only the training target, the authors isolate the value of textual critique. The finding that CAFT-Critique beats CAFT-Correction across all eight student-environment pairs demonstrates that the critique itself carries actionable supervision, not merely that keeping the mistake in context helps. The analysis of expert requirements is also well-executed, showing that privileged task information matters far more than elaborate prompt engineering.
Generalizability and overstated auxiliary claims are the primary weaknesses. The RL warm-start experiment is a single case study on one model and one environment, making it impossible to conclude that CAFT checkpoints generally outperform distillation for RL initialization. Iterative CAFT shows mixed returns—Gemma 4 31B actually degrades on $\tau^{2}$ AR after a second pass—yet the paper still presents repeated fine-tuning as a promising direction without diagnosing the degradation. Moreover, three of the four students are from the same Gemma 4 family as the expert, raising questions about whether critique quality and student comprehension are artificially aligned; the lone cross-family student (Nemotron 3 Nano 4B) sees the largest gains, but the sample size is small.
The evidence solidly supports the superiority of critique-augmented over action-only supervision, but comparisons to alternative critique-based methods are limited to related-work discussion rather than head-to-head benchmarks. The paper fairly notes that expert-trajectory distillation can collapse when the student mimics exploration without learning stopping rules, yet it does not test whether CAFT's advantage persists against stronger baselines such as on-policy distillation with dense token-level feedback. The expert-requirement ablation is a genuine strength, clearly showing that gold reference actions and evaluation criteria are more important than detailed critique prompts.
The authors provide comprehensive implementation details, including full expert prompts in Appendix B, training hyperparameters in Appendix C, and GRPO settings in Appendix D. Both benchmarks are publicly known, and the training procedure is described with sufficient granularity to permit independent replication in principle. Nevertheless, no code or datasets are released, and the $\tau^{2}$-Bench evaluation pipeline depends on a Gemma 4 31B-based user simulator and judge, which means reproducing the exact numbers requires access to that specific model. The reported hyperparameters such as $\lambda=0.1$ were selected via an internal sweep that is only partially documented.
Large language models (LLMs) are increasingly used for multi-turn agentic tasks that require interactions with users and environments over extended sequences of actions. A common approach to improving LLM agents is to use more capable expert models to provide training supervision, either by behavior cloning the expert's trajectories or by using the expert to correct the actions in trajectories generated by the weaker student model. However, expert LLMs can provide richer supervision than actions alone: they can directly generate textual critique that explain why specific actions are incorrect. We study whether training on such feedback benefits LLM agents and propose Critique-Augmented Agentic Fine-Tuning (CAFT), a framework that elicits textual expert feedback on student-generated trajectories. For each incorrect action, the expert yields both a corrected action and a textual critique explaining the student's mistake--which we then use as the training target for fine-tuning. Across four student models from the Nemotron 3 and Gemma 4 families on the agentic domains $\tau^2$-Bench Airline+Retail and WebShop, adding critiques to action-level supervision improves performance by 8% on average over action-only distillation. Additionally, CAFT checkpoints can also provide a better warm start for reinforcement learning than expert-trajectory distillation. Our results demonstrate that textual critiques can strengthen supervision for multi-turn LLM agents.
Pick a starting point or write your own. Challenges run in the background, so you can keep reading while the AI investigates.
No challenges yet. Disagree with the review? Ask the AI to revisit a specific claim.