Co-Evolving Acting and Forecasting for Language Agents
CoPE attacks the sample-efficiency bottleneck in training long-horizon language agents with reinforcement learning. Standard agentic RL such as GRPO optimizes only a sparse terminal reward $Y(\tau) \in \{0,1\}$, leaving the rich temporal structure of intermediate steps unused. The paper introduces CoPE, which augments GRPO with success-conditioned multi-step action forecasting: on successful on-policy trajectories, the shared policy backbone learns to predict future action chunks $a_{t:t+K}$ from each history $h_t$, so every action is supervised from both the moment it is taken and from the $K$ preceding histories. This implicit temporal ensemble yields stronger aggregate performance and up to $2\times$ sample-efficiency gains across five benchmarks, with no added inference cost.
CoPE is a clean, well-motivated extension of GRPO that generally improves aggregate performance on simulated and realistic agent benchmarks. Its core idea—replacing noisy observation targets and shared scalar value targets with concise, step-specific action chunks—is intuitively appealing and backed by ablations. Empirically, it delivers large gains on AppWorld (55.36% vs. 41.67%) and ScienceWorld (72.5% vs. 64.5%). However, the paper also reveals a clear failure mode on the Airline domain of $\tau^2$-bench, where success filtering skews the policy toward easy suboptimal behavior. The theoretical framing in Sec. 4 provides useful intuition but explicitly idealizes the GRPO update, so the formal justification is partially decoupled from the implemented algorithm.
The paper’s central insight—that multi-step action chunks provide decision-relevant, step-specific supervision compared to noisy observations or uniform scalar rewards—is well supported by both theory and ablation. Table 5 shows that moving from delay-0 imitation to a full delay ensemble ($K=2$) raises ScienceWorld success by 6.0 points over the $K=0$ baseline, confirming that the temporal ensemble matters. The separated-forecaster experiment in Sec. 8 is particularly compelling: adding an external LoRA forecaster to rerank actions improves GRPO but adds nothing to CoPE, which implies the shared backbone has already internalized the ensemble. Wall-clock costs are also genuinely small: the forecasting pass adds 11% per-step time, and shorter rollouts net a 9% end-to-end reduction versus GRPO.
First, the theoretical analysis in Sec. 4 assumes an idealized on-policy, unclipped GRPO to derive Eq. (3), while Algorithm 1 uses the standard off-policy clipped surrogate with KL penalty. The authors candidly note this gap, but the paper does not quantify how clipping or KL affect the success-conditioned action-chunk target. Second, CoPE inherits a severe pathology of outcome-conditioned imitation: when easy successes dominate the filtered trajectories, the forecaster learns suboptimal behavior that transfers to harder look-alike tasks. This is not a minor corner case—it directly explains the Airline collapse (26.2% down to 16.2%), where refusing service (the easy success mode) propagates into tasks that actually require action. GRPO avoids this because failed refusals receive negative advantages, whereas CoPE’s forecasting loss learns only from successes.
Third, the method assumes actions are short and decision-relevant relative to observations. In domains where actions themselves are long structured outputs—such as large code diffs or lengthy API call sequences—the advantage of action targets over observation targets may shrink, and the paper does not test this boundary.
The evidence is strongest on the three simulated benchmarks, where CoPE outperforms ECHO and RWML on aggregate metrics. However, the comparison is asymmetric: ECHO predicts single-step observations, while CoPE predicts multi-step action chunks, so the baseline does not isolate whether the gain comes from target modality, horizon length, or both. On realistic benchmarks the picture is mixed. AppWorld shows a large controlled improvement over in-house GRPO, but published G2PO numbers use a different subset and protocol and are therefore incomparable. On $\tau^2$-bench the overall average barely moves (27.5% to 28.7%) and is propped up by Retail while masking an Airline regression, which weakens the claim of universal gains. A stronger experimental design would include a multi-step, success-filtered observation-prediction baseline to isolate the value of action targets.
The algorithm, hyperparameters (Appendix B), and prompts (Appendix C) are documented in reasonable detail. Yet there is a direct contradiction: the abstract claims the codebase is already available on GitHub, while the Reproducibility Statement says full configuration files and training code will be released only upon publication. This ambiguity is a needless obstacle. Exact reproduction of AppWorld and $\tau^2$-bench results also depends on proprietary or stochastic environment backends (e.g., GPT-4o-mini customer simulator, AppWorld scenario engine) whose versions and API behavior are not pinned, and the paper does not report random seeds or total compute budget.
Training language agents is bottlenecked by the high cost of long-horizon rollouts, yet standard agentic reinforcement learning (RL) learns only from a terminal scalar reward, leaving the supervision in intermediate steps largely unused. Prior work exploits this supervision through value targets that estimate per-step success probabilities or observation targets that fit environment response. Both are hard to construct well: (1) value targets come from the same single scalar signal for each step in the trajectory, and (2) observation targets, such as web pages and terminal logs, are long and noisy and reach only one step ahead. We propose \name, which adds a third kind of multi-step action targets. On successful on-policy trajectories, \name trains the policy backbone to generate the action sequences that follows each history. Each action is also supervised when forecast from the preceding histories, forming a temporal ensemble that the shared backbone absorbs during training, and this enables acting and forecasting to co-evolve. The policy still acts one step at a time at inference, with no overhead. Across five benchmarks (ALFWorld, ScienceWorld, WebShop, $\tau^2$-bench, and AppWorld), \name (1) achieves stronger aggregate performance than GRPO and RL baselines with predictive auxiliary targets across simulated environments, while improving overall performance on both realistic applications; (2) delivers up to $1.9\times$ and $2.0\times$ higher sample efficiency than ECHO and GRPO, respectively, on ScienceWorld; and (3) achieves these gains with a single policy backbone, without additional models, separate pretraining, or inference overhead. The codebase is available at https://github.com/ulab-uiuc/CoPE.
Pick a starting point or write your own. Challenges run in the background, so you can keep reading while the AI investigates.
No challenges yet. Disagree with the review? Ask the AI to revisit a specific claim.