Explorer-Zero: Improving Self-Play for Long-Horizon Tasks with Decomposable Verifiable Rewards
Explorer-Zero tackles the data-curation bottleneck for agentic RL by training a learned proposer to explore stateful environments, compose multi-step tasks from demonstrated transitions, and record intermediate environment states as verifiable checkpoints. These checkpoints supply fine-grained rewards $R^{\text{subgoal}}$ that prevent group-normalized advantages from collapsing when the solver fails, while alternating proposer–solver training adapts the curriculum automatically. On ALFWorld, ScienceWorld, and AppWorld, the approach outperforms strong self-play baselines and narrows the gap to human-data RL.
The paper presents a well-motivated and empirically strong self-play framework. The idea of reusing the proposer trajectory as both compositional tasks and state-based verification artifacts is natural and effective, yielding large gains over prior self-play methods. However, the theoretical contribution is a basic conditional-probability observation rather than a deep RL insight, and the method is largely restricted to resettable simulators with hand-coded state-matching rules.
The ablation in Table 2 cleanly isolates the contributions of interactive proposal and subtask rewards. Adding subtask supervision yields further gains of 18.58 points on ALFWorld and 13.84 TGC points on AppWorld beyond interactive proposal alone, showing that fine-grained rewards provide substantial value. The analysis in Section 5.4 is compelling: benefits concentrate on high-complexity tasks, and self-play initialization improves subsequent human-data RL across all three benchmarks.
Proposition 1 is a straightforward discrete-probability tautology: it merely notes that if failed rollouts reach different subtask counts, normalized advantages remain non-zero, yet the paper concedes in Appendix B that a failed rollout completing every subtask receives the same reward as success, which can create a local optimum near full subtask completion.
The verification mechanism relies on brittle state-delta matching ($s \simeq s^{\text{ref}}$) with hand-tuned rules per environment, and Appendix H explicitly acknowledges this limits deployment to resettable simulators rather than live systems. Evaluation variance is also nontrivial with only four samples per task.
Comparisons to Self-Challenging and R-Zero are fair and show consistent margins across benchmarks. The per-complexity breakdown in Figure 4 supports the claim that subtask rewards help most on multi-step tasks. However, the paper does not directly disentangle task-distribution quality from reward granularity: the Table 2 ablation changes both task construction and solver rewards simultaneously, and the authors note it 'does not isolate reward granularity alone.'
Hyperparameters, prompt templates, and the alternating training algorithm are documented in Appendices D, G, and C. However, no code release or pretrained checkpoints are mentioned, and reproducing the method would require implementing environment-specific state-matching rules and validity filters that go beyond standard RLVR pipelines.
Training LLM agents on long-horizon interactive tasks with reinforcement learning typically relies on human-annotated tasks and reliable verification, both expensive to construct for stateful environments. Self-play offers an alternative, but must generate achievable, compositional tasks and provide feedback beyond binary outcomes. We propose Explorer-Zero, a self-play framework that jointly generates tasks and fine-grained, verifiable rewards through environment interaction. A learned proposer explores the environment, constructs tasks, and records intermediate and final states as verification targets. These checkpoints reward partial completion, while alternating proposer--solver training adapts the generated curriculum. We show theoretically that subtask rewards reduce collapse among unsuccessful rollouts when they complete different numbers of subtasks. Without human-annotated training tasks, Explorer-Zero outperforms the evaluated self-play baselines across ALFWorld, ScienceWorld, and AppWorld; on AppWorld, our 30B agent reaches 59.67% task success, approaching 62.65% for RL on human-annotated tasks. Further analyses support task-relevant composition and benefits on complex tasks, while self-play initialization improves subsequent human-data RL across all three benchmarks.
Pick a starting point or write your own. Challenges run in the background, so you can keep reading while the AI investigates.
No challenges yet. Disagree with the review? Ask the AI to revisit a specific claim.