ReflectPO: Reflection-Guided Policy Optimization for Long-Horizon LLM Agents
ReflectPO tackles autonomous error recovery in long-horizon LLM agents, where early execution mistakes compound across sequential tool-use interactions and existing RL methods optimize task outcomes without teaching the policy when and how to self-correct. The core idea is to generate recovery trajectories from rollback snapshots, guided by reflection hints derived from branch differences across rollout trees and repeated loops within trajectories, and then optimize the policy with tree-structured credit assignment that captures both local decision effects and recovery gains. This matters because preliminary evidence shows that errors are nearly ubiquitous in natural trajectories yet agents autonomously recover in fewer than 15% of cases, leaving substantial headroom for training-based correction.
ReflectPO is a conceptually clean and empirically solid contribution to long-horizon agent RL. Its reflection-guided sampling and tree-structured credit assignment yield consistent overall gains on BFCL-v3 and $\tau^2$-Bench, and the ablation supports the value of recovery-aware training. However, its heavy reliance on a powerful external guide model for hint generation and on restorable environment snapshots narrows applicability, and the near-zero incremental benefit of combining both sampling strategies suggests the two components overlap more than the framing implies.
The paper makes a strong empirical case that autonomous recovery is the central bottleneck in long-horizon execution: preliminary evaluation with a judge model shows error trajectory rates above 79% and complete recovery rates near 14% on natural rollouts, with step recovery at only 18.74%. The proposed credit assignment is well-motivated, building directly on RTMC by using parent–child reward differences $A^{\text{raw}}(v,a)$ in the prefix tree, and the recovery analysis shows ReflectPO improves both step-level and complete recovery over a strong Reinforce++ baseline. The BFCL-v3 Multi-Turn gains are pronounced and consistent across model sizes.
A critical limitation is the dependence on DeepSeek-V4-Pro as an external guide model to generate reflection hints during training; because baselines do not appear to receive comparable teacher guidance, the comparison partly conflates algorithmic innovation with privileged distillation. The ablation in Table 3 reveals that combining contrast-based and loop-based sampling adds only 0.09 overall points over contrast-only, suggesting the strategies overlap in utility despite the narrative framing them as complementary. Additionally, the evaluation of error recovery relies entirely on GPT-5.6 Terra as an automated judge without validated human alignment or inter-rater agreement. The method also requires saveable and restorable environment snapshots to resume execution at recovery anchors, which is not always practical in real-world tool-use deployments.
The main results in Table 2 largely support the headline claim that ReflectPO achieves the best overall average at both 4B and 8B scales, with especially strong gains on BFCL-v3 Multi-Turn Long-Context. However, improvements are uneven across subdomains; for example, on Qwen3-8B $\tau^2$-Bench Retail, GiGPO outperforms ReflectPO 50.0 to 46.5, and on Qwen3-4B Retail GiGPO also leads 48.3 to 44.7. Comparisons to prior work are fair in terms of shared data and model sizes, but unequal access to a strong teacher model during training complicates the claim that the gains come purely from better RL credit assignment rather than from high-quality hint augmentation.
The paper provides implementation details in Appendix A.1—batch size 32, three epochs, snapshot window lengths, and KL coefficient 0.001—but omits core optimization hyperparameters such as learning rate, optimizer, and learning-rate schedule. Code is not released. Reproduction would be blocked by the need for DeepSeek-V4-Pro API access to generate reflection hints and to simulate users, as well as by the requirement for restorable environment snapshots that capture full internal state. Network fluctuations during evaluation can cause task failures, which are excluded from performance calculations, potentially introducing upward bias. Data inspection and pruning left only 2457 training samples, raising stability concerns for RL.
With external tools, large language models (LLMs) can tackle complex tasks through sustained environmental interaction. LLM-based agents are therefore increasingly used to tackle complex long-horizon tasks in real-world environments. However, these tasks involve sequential, interdependent decisions and changes in environmental states. Early execution errors affect subsequent interaction histories and environmental states, and their effects accumulate along the decision chain. To address this challenge, we propose Reflection-Guided Policy Optimization (ReflectPO), a reinforcement learning framework for autonomous error recovery in long-horizon tasks. ReflectPO comprises two core components. The first is Reflection-Guided Sampling. We use branch differences and repeated visits in interaction trajectories to construct hints for error recovery. The second is Tree-Structured Credit Assignment. We organize sampled trajectories into trees and assign action-level credit using parent--child reward differences and recovery gains. Experiments over two long-horizon tool-use benchmarks show that ReflectPO outperforms multiple existing baselines in most domains. It also achieves the best overall performance on average. Further analyses show that ReflectPO effectively improves agents' ability to recognize their own deviations, recover from erroneous states, and continue making progress toward task completion.
Pick a starting point or write your own. Challenges run in the background, so you can keep reading while the AI investigates.
No challenges yet. Disagree with the review? Ask the AI to revisit a specific claim.