Where to Start, How to Move: Analyzing Curricula under GRPO for Long-Horizon LLM Agents
GRPO for long-horizon agents suffers from sparse terminal rewards that yield all-fail rollout groups and zero advantage signals early in training. This paper studies guide-assisted curricula that replay successful reference prefixes before the learner continues, systematically varying the initial position, handoff granularity, and movement rule. Across Qwen3 models from 1.7B to 8B on BFCL and AppWorld, the authors find that starting near task completion and using outcome-informed call-level selection improves independent execution while reducing training cost. The work usefully frames curriculum design as a practical lever for GRPO-based agent post-training.
The paper presents a clearly scoped empirical analysis of curriculum design for GRPO agent training, with systematic ablations over initialization, handoff unit, and movement rule. The core result—that End initialization outperforms Start and that outcome-informed call-level bandits improve over GRPO—is supported across three model sizes and two benchmarks. However, the study is restricted to a single model family, lacks independent training-run variance, and confounds handoff granularity with movement-rule differences, leaving several claims weaker than stated.
The End-initialization benefit and the reduction in all-fail training groups are the strongest findings. At 8B, Turn-Threshold reduces the all-fail share from 43% to 26% early in training and to 13% by the end, while Call-Bandit reaches 12%, both substantially below GRPO's 23%. Call-Bandit also reaches the highest validation accuracy during training, and its gains transfer to AppWorld interactive coding.
The comparison between Turn and Call handoff units is confounded by differing step sizes and outcome-history settings, so the claim that finer handoffs alone do not help is not cleanly isolated. The DAPO comparison at a fixed generated-rollout budget is arguably unfair because DAPO is designed to consume extra rollouts to replace degenerate groups; capping total rollouts handicaps its core mechanism. Furthermore, evaluation relies on three output samples per task but no independent training runs, so margins such as 55.0 versus 57.0 Avg@3 may not be robust to seed. Finally, the paper does not compare against stronger baselines such as PPO with a learned value function or simple intermediate reward shaping.
The evidence supports the headline claim that guided starts improve GRPO training, but some effect sizes are small and a cross-category performance reversal between Turn-Threshold and Call-Threshold is left unexplained. Call-Bandit leads in Base and Macro Avg@3 across all sizes, yet Call-Threshold outperforms Turn-Threshold on untrained categories after underperforming on Base, suggesting that generalization dynamics are more nuanced than the main narrative implies. Comparisons to related work are brief but fair; the paper adequately positions itself against JSRL, R3, and Agent-G2 without overstating novelty.
Training uses public models and benchmarks with reported hyperparameters (AdamW, lr $10^{-6}$, weight decay $0.01$, 16 continuations per group), but no code, checkpoints, or random seeds are released. The provenance of successful guide trajectories is unspecified, and because call-level positions evolve differently across methods, reproducing the curriculum requires replicating an internal Thompson-sampling selector and replay logic. Independent reproduction is therefore possible in principle but practically blocked by missing implementation details and the absence of cross-run variance estimates.
Group Relative Policy Optimization (GRPO) provides no reward-relative learning signal when all rollouts receive identical rewards. In long-horizon tasks that require multiple interactions, rewards are given only upon successful task completion, making all-fail groups common early in training and limiting opportunities to practice later steps. Guide Assisted GRPO replays a successful reference prefix before the learner continues. We study how initial positions, handoff units, and movement rules affect unassisted task completion. Across three model sizes on BFCL, starting near completion is more effective under a fixed turn-level threshold rule. Finer handoffs alone do not consistently help, but outcome-informed tool-call selection outperforms GRPO on trained and untrained task categories. Guided starts also improve AppWorld success on both evaluation splits. These findings highlight where learning starts and how starting positions move as key curriculum choices.
Pick a starting point or write your own. Challenges run in the background, so you can keep reading while the AI investigates.
No challenges yet. Disagree with the review? Ask the AI to revisit a specific claim.