Reusing Past Samples in Proximal Policy Optimization: When and How Does It Help?
PPO is the workhorse of on-policy deep RL, yet it discards collected data after just a few updates, leaving substantial sample efficiency on the table. This paper asks whether reusing recent off-policy samples within a multiple importance weighting framework can improve PPO without changing its core mechanics. The authors propose two variants—uniform ($\omega$PPO-U) and balance-heuristic-corrected ($\omega$PPO-BH)—derive policy improvement bounds that account for stale advantages, and empirically show that moderate windowed reuse boosts final performance while excessive reuse can backfire.
The paper delivers a clean, controlled study of sample reuse in PPO. By deriving lower bounds for both uniform and BH-corrected importance weighting, the authors formalize the intuition that stale advantages and policy drift impose limits on how far back one should reuse. Their key empirical finding—that benefits come from the extra data rather than simply more gradient steps—is well-supported across MuJoCo tasks. However, the analysis stops short of providing actionable, automatic rules for choosing the window size $\omega$, and the evaluation is confined to standard continuous-control benchmarks. As the authors summarize, "the benefit stems from from the additional data rather than from the increased number of updates alone" (Section 4). The bounds also reveal uncontrollable TV-divergence penalties, "suggesting that the policies in the reuse window should remain close to each other" (Section 5).
The experimental design is the strongest aspect. The authors run two carefully calibrated comparisons—fixed mini-batch size and fixed mini-batch count—which isolate the value of reused data from the confounding effect of additional updates. They also deliberately avoid re-evaluating past advantages or modifying PPO's clipping, keeping the study focused. The result is convincing evidence that simple cross-iteration reuse can improve final deployed performance. The authors' framing is apt: they set out to conduct "a controlled study of the effects of sample reuse in PPO" by "deliberately preserving its core mechanics to isolate the effect of data reuse from other factors" (Section 5).
Despite the theoretical grounding, the bounds contain penalty terms independent of the optimization variable that do not translate into practical algorithmic guidance beyond the generic advice to keep policies close. Empirically, the evaluation is limited to MuJoCo, leaving transferability to discrete or high-dimensional domains (e.g., LLM fine-tuning) unresolved. Moreover, the window size $\omega$ behaves like a sensitive hyperparameter: on Hopper-v5, "at $\omega=8$, the reuse variants not only fail to outperform PPO but exhibit degraded training" (Section 4). The authors conjecture that stale advantages, a diminished fraction of fresh data, and increased policy drift are to blame, yet offer no mechanism to adapt $\omega$ online or per-task. Finally, neither weighting scheme dominates, and the BH variant imposes $2(\omega_k-1)D$ extra policy evaluations per step for inconsistent gains, raising questions about its cost-benefit trade-off.
The internal evidence supports the central claim that moderate reuse improves final performance, but the paper deliberately avoids comparing against off-policy baselines or recent hybrid methods such as GePPO or SARD-PPO. While this isolation is methodologically sound, it leaves readers without a sense of absolute sample-efficiency gains relative to SAC or TD3. The authors note that prior sample-reuse variants "require re-evaluation of stored samples, rely on heuristics or modified clipping mechanisms, and are presented as standalone methods ... a systematic investigation of when, how, and why data reuse in PPO is beneficial remains unaddressed" (Section 1). The fixed-$N_{\mathcal{B}}$ comparison is particularly compelling because it equalizes fresh data per update, making the reuse signal unambiguous.
The experiments adopt the tuned Stable-Baselines3 PPO hyperparameters and standard MuJoCo environments, with thorough details in Appendix C. However, the manuscript does not mention publicly releasing code. Reproducing the mini-batch construction—each mini-batch must equally represent the full composite dataset $\mathcal{D}(\omega_k)$—and the balance-heuristic ratios requires careful implementation. In particular, "evaluating the BH-corrected loss requires evaluating all samples in $\mathcal{D}(\omega_k)$ under all policies $\{\pi_{\bm{\theta}_{k-i}}\}_{i=0}^{\omega_k-1}$ in the window" (Section 3.2). Without released code or exact hyperparameter schedules, independent reproduction is feasible but nontrivial, especially for the BH variant.
Among on-policy deep reinforcement learning methods, Proximal Policy Optimization (PPO) has become the de facto standard, due to its consistently strong empirical performance across diverse application domains. However, on-policy methods are inherently sample inefficient: fresh data collected under the current policy is used for just a few updates before being discarded. Off-policy methods avoid this inefficiency via experience replay, achieving notable sample efficiency gains, but at the cost of training instabilities or extensive tuning. This motivated the rise of hybrid strategies that augment PPO with off-policy data reuse. Existing sample-reuse variants of PPO demonstrated improved sample efficiency over vanilla PPO, yet a systematic study of when reuse helps, in which scenarios, and to what extent remains missing. In this work, we study the effectiveness of sample reuse in PPO by instantiating two variants within a multiple importance weighting framework. Both retain the core PPO mechanics, reusing only samples from a window of recent iterations, thereby isolating the effect of data reuse from other factors. The variants, termed wPPO-U and wPPO-BH, employ vanilla importance weights or balance-heuristic-corrected ones, respectively. For both, we derive policy improvement lower bounds providing theoretical grounding for their respective losses. We use them to empirically study when and how data reuse improves sample efficiency or final performance of PPO across continuous control tasks.
Pick a starting point or write your own. Challenges run in the background, so you can keep reading while the AI investigates.
No challenges yet. Disagree with the review? Ask the AI to revisit a specific claim.