Long-Horizon RL Needs Muon
This paper studies policy optimization for long-horizon software-engineering agents that learn from sparse binary test outcomes. The core idea is a training recipe—dubbed SPO++—that combines the Muon optimizer with single-rollout asynchronous updates, truncated importance sampling (TIS), token-averaged loss, and auxiliary length/tool-error penalties. The authors claim this combination achieves higher training success than SPO baselines on repository-level issue resolution for two model families, suggesting Muon is critical for long-horizon RL.
The paper presents an interesting engineering recipe, but its central claim is overstated and under-supported. The title asserts that long-horizon RL 'needs' Muon, yet the evidence is confined to a single task (repository-level issue resolution on R2E-Gym), two models, and a small number of explicitly 'selected' training runs. Crucially, the authors' own concurrent work (Ruan et al., 2026a) shows that Muon's advantage over AdamW largely vanishes under RMS-matched controls and that tuned AdamW nearly matches Muon at scale, directly contradicting the universal framing here. The full recipe changes the optimizer, loss aggregation, importance correction, clipping, and reward shaping simultaneously, so the causal attribution to Muon is unsupported.
The component ablations are useful. The paper isolates the effects of token-level versus sequence-level loss aggregation, TIS, and auxiliary penalties, and reports that several Adam configurations decline while the full recipe improves in the evaluated runs. The observation that asynchronous single-rollout training benefits from combining importance correction with per-token averaging is a practical finding for the SWE-agent setting.
First, the title and framing make a universal claim from extremely narrow evidence. Only two models are tested on one task family, and the metric is training success rate on rollouts from the training distribution rather than generalization on a held-out benchmark. Second, the paper explicitly relies on 'selected' runs, including a 'separately selected run' for the SPO + Muon baseline, which invites cherry-picking. Third, there is no statistical testing: individual training curves (thin translucent lines in Figures 4 and 5) suggest very few seeds per configuration, yet no confidence intervals or aggregated statistics are provided. Finally, the companion paper by the same authors establishes that Muon's gains are tied to its larger effective step size and largely disappear when that scale is controlled, undermining the causal narrative that Muon's spectral properties are uniquely beneficial.
The evidence supports only a narrow, correlational conclusion: the specific combination of ingredients in the full recipe achieved higher training success than the baselines in the runs that were selected for display. It does not support the stronger claim that Muon is necessary. The comparisons to SPO and SPO++ are fair insofar as they hold data budgets fixed, but the full recipe changes multiple hyperparameters and algorithmic components at once, so the marginal contribution of Muon is confounded. Related work on Muon in RLVR (e.g., Fan et al., 2026) points to spectral whitening failures that the current paper does not address, and the authors' own prior work confirms these regime-dependent effects.
Reproduction would be difficult. The paper does not release code or provide exhaustive hyperparameters for the Muon implementation, SPO++ details, or the asynchronous rollout infrastructure beyond citing the Claude Code harness version. The number of independent runs per configuration is not stated, and the explicit selection of runs means readers cannot know how many runs were filtered out or on what basis. Key hyperparameters such as the Muon learning rate, Newton–Schulz iteration count, and momentum coefficients are omitted, as are the precise TIS $C=2$ clipping rationale and the protocol for handling 'infrastructure failures.'
Long-horizon agents must coordinate reasoning, tool use, and environment feedback over extended interactions. Learning from sparse outcome rewards requires weighting trajectories of different lengths and updating the policy while rollouts are still being collected. We examine a single-rollout asynchronous recipe combining Muon, token-averaged loss, truncated importance sampling, and auxiliary length and tool-error penalties. Single-rollout sampling allows asynchronous learning without waiting for multiple trajectories for the same issue. We study optimizer choices and recipe components for repository-level issue resolution across two model families. The evaluated training runs show higher training success with the full recipe than with the baselines.
Pick a starting point or write your own. Challenges run in the background, so you can keep reading while the AI investigates.
No challenges yet. Disagree with the review? Ask the AI to revisit a specific claim.