FERPO: Forward Entropy-Regularized Policy Optimization
Policy-gradient methods suffer from high variance, while pathwise actor updates require accurate action derivatives that standard value-function training does not guarantee. FERPO addresses this by deriving an entropy- and KL-regularized target action distribution from critic values alone, then fitting the policy with forward KL via self-normalized importance sampling---never differentiating the critic with respect to actions. The method targets massively parallel on-policy robot learning, where reliable policy improvement is critical for both sample and wall-clock efficiency.
FERPO is a principled on-policy maximum-entropy actor-critic method that replaces critic action gradients with a value-based forward-KL target fit using SNIS. Its theoretical derivation is clean and the ESS-based trust-region control is practically motivated. However, the empirical advantage over its closest competitor REPPO is uneven: while FERPO offers faster actor updates and slightly better sample efficiency on some benchmarks, it underperforms REPPO on the G1/T1 locomotion suite. The Bonferroni-corrected bootstrap gives an IQM difference of -1.41 with adjusted 95% interval [-1.94,-0.94], entirely favoring REPPO.
The derivation of the optimal regularized target from the Lagrangian provides a principled way to combine value weighting with policy proximity. The bimodal toy experiment clearly illustrates the mechanism: at a reference mean $\mu=-0.6$, the reverse-KL direction used by REPPO moves left while FERPO's forward-KL fit moves right toward the better mode, since $\mathbb{E}_{\pi_\mu}[c'] = 0.284$ and $\mathbb{E}_{q_\mu^*}[c'] = -0.0419$.
The most significant empirical limitation is that FERPO trails REPPO on the G1/T1 locomotion tasks, which are central to the paper's robotics motivation: the adjusted bootstrap interval for the suite IQM is entirely below zero, and REPPO's aggregate return of $34.63\pm0.17$ exceeds FERPO's $33.22\pm0.28$. Importance-sampling variance is a practical bottleneck: even with K=1,024 samples, the rollout policy recovers the correct direction in the bimodal problem in only 26.4% of batches. The theoretical trust-region bound (Proposition 1) also assumes exact population fitting and exact critics that do not hold with neural function approximation, and GPU memory usage is 22.4-25.6% higher than REPPO.
The evidence supports competitive performance but does not establish uniform dominance. On DMC, FERPO has the highest IQM point estimate of $897.7\pm6.7$ versus REPPO's $891.0\pm6.4$, yet the Bonferroni-corrected 95% interval for the difference is $[-6.89,20.65]$, spanning zero and making the lead statistically inconclusive. On ManiSkill, the two methods are virtually tied ($95.67\%$ vs. $95.52\%$). On G1/T1, REPPO wins decisively. Comparisons to PPO and FastTD3 are fair but these are not the primary algorithmic target.
The authors provide public code, detailed pseudocode in Algorithm 1, and extensive protocol descriptions, which is commendable. Reproduction would still face hurdles: hyperparameters differ across experiments---the G1 candidate-count ablation uses normalized ESS target 0.85 while the G1/T1 benchmark uses 0.65---and peak GPU memory is 22.4% higher for uncached FERPO and 25.6% higher for cached FERPO than for REPPO. The method's performance also depends strongly on critic capacity; both methods collapse when critic width drops to 62.
Several state-of-the-art methods for online reinforcement learning in continuous control improve policies using action gradients of a learned critic. However, critics are typically trained to predict returns, and accurate value predictions do not necessarily yield accurate action derivatives, potentially leading to unreliable policy updates. We propose Forward Entropy-Regularized Policy Optimization (FERPO), an on-policy maximum entropy reinforcement learning algorithm that performs policy improvement using critic values without differentiating the critic with respect to actions. FERPO derives an optimal target action distribution from a policy-improvement objective regularized by entropy and Kullback-Leibler (KL) divergence. We then fit the actor to this target by minimizing a forward-KL objective, estimated using self-normalized importance sampling (SNIS) with actions drawn from the rollout policy. By limiting the target distribution's deviation from the rollout policy, the KL regularization helps keep these importance weights well behaved. In contrast to reverse-KL objectives, which can favor a subset of the target distribution's modes, the forward-KL objective encourages coverage of multiple high-value modes and thereby promotes exploration. Experiments and ablations on MuJoCo Playground and ManiSkill show competitive performance and sample-efficiency gains. Computational benchmarks also demonstrate faster actor updates than Relative Entropy Pathwise Policy Optimization (REPPO).
Pick a starting point or write your own. Challenges run in the background, so you can keep reading while the AI investigates.
No challenges yet. Disagree with the review? Ask the AI to revisit a specific claim.