Continual Reinforcement Learning with Neuroevolution
Continual reinforcement learning demands a delicate balance between plasticity (learning new tasks) and stability (not forgetting old ones). This paper argues that neuroevolution—specifically evolution strategies (ES) and genetic algorithms (GAs)—offers a compelling alternative to gradient-based policy-gradient methods, which tend to lose plasticity as tasks change. Across a broad sweep of environments and task-shift types, the authors show that ES consistently achieves the best stability-plasticity trade-off, and they propose a clear mechanism: parameter-space exploration produces wider "neighborhoods" of robust solutions that overlap across sequential tasks. The work matters because it challenges the prevailing focus on fixing gradient-based RL and instead revives a classic optimization paradigm with strong empirical and conceptual justification.
This is a strong, well-executed study that makes a persuasive case for neuroevolution in continual RL. The breadth of environments (from 386-parameter CartPole to >1M-parameter Kinetix), the explicit measurement of return landscapes, and the correlation between neighborhood overlap and stability-plasticity trade-offs give the paper both empirical weight and theoretical texture. The authors are appropriately careful: they acknowledge where RL still wins (HalfCheetah) and where their methods need heuristic fixes.
The paper’s central claim—that ES achieves the best stability-plasticity trade-off—is backed by extensive experiments spanning classic control, sparse-reward navigation, locomotion, and pixel-based physics puzzles. The authors do not just report aggregate metrics; they visualize return landscapes and show that the shared neighborhood fraction correlates with LA − F across families of methods. The decision to compare against state-of-the-art continual RL variants (TRAC, ReDo, C-CHAIN) and population-based RL (PBT-PPO) rather than just vanilla PPO makes the comparison demanding and fair. The code is released, which greatly strengthens the work.
Several limitations temper the paper’s conclusions. First, NE is acknowledged to be less sample-efficient and to require large populations; the experiments use 512 candidates per generation for ES and the GA, yet the paper does not fully quantify the sample-efficiency penalty relative to gradient-based methods in this continual setting. Second, on HalfCheetah—a challenging continuous control task—RL methods outperform NE even in the stationary case, suggesting NE does not reliably scale across all problem types. Third, the GA relies on a hand-tuned heuristic to force its centroid to track the elite on MountainCar and Kinetix, without which the centroid fails entirely; this introduces an asymmetry not present in the baselines. Fourth, the novelty-search experiments are limited to only two specialized settings (MountainCar with stronger noise and DeepSea), so the claim that diversity makes populations "more plastic" is less general than the rest of the study. Finally, while LA − F is a standard metric, the paper rescales LA and F per setting, which can obscure absolute performance gaps when comparing across environments.
The evidence broadly supports the main claims, but some comparisons deserve scrutiny. The continual RL variants (TRAC, ReDo, C-CHAIN) restore plasticity relative to PPO but do so at the cost of stability, which is consistent with prior literature. However, the authors note that PBT-PPO’s advantage comes mainly from reduced per-agent updates rather than maintained population diversity, which highlights that the population mechanism alone is insufficient. The neighborhood-width analysis is persuasive, though it measures robustness to random perturbations rather than task-specific gradients, and the authors partially address this by also analyzing neighborhoods along each method’s actual update path. Related work is thorough and fairly contextualizes the contribution within both continual RL and evolutionary computation.
Reproducibility is solid but not perfect. The authors publish their codebase and provide detailed hyperparameter tables for every method and environment. Pseudocode for ES, GA, and all RL variants is included in the appendix. However, the GA’s centroid-tracking heuristic on MountainCar and Kinetix was hand-tuned in short pilots without a systematic search, which complicates independent reproduction of the GA results on those two environments. Additionally, while the main text says compute is matched, the exact FLOP or wall-clock accounting between population-based NE and gradient-based RL is not itemized, so a reproduction would need to carefully verify the generational update budgets.
Despite many studies about causes and remedies of plasticity loss in Reinforcement Learning (RL) under continual task changes, no RL method has yet consistently achieved a good balance between adaptation and forgetting. Here we turn to an alternative optimization paradigm, neuroevolution (NE): algorithms that search directly in weight space through mutation and selection over a population of neural networks. Across a wide array of environments and environmental changes, with policies ranging from a few hundred parameters to million-parameter networks, we compare evolution strategies (ES) and genetic algorithms (GAs) against state-of-the-art continual RL variants and population-based RL. ES most consistently achieves a good stability-plasticity trade-off, while the GA is the most plastic method but forgets more than ES. To explain this, we study the return landscape around each method's solutions. ES finds the widest neighborhoods, i.e.\ regions of weight space in which perturbed policies still solve the task, and the size of the overlap between the neighborhoods of consecutive tasks correlates with a method's stability-plasticity trade-off. Rewarding behavioral diversity in a GA through novelty search makes the population even more plastic, at the cost of forgetting. Finally, symptoms of plasticity loss commonly reported in RL do not transfer to NE. Overall, these results establish NE as a competitive alternative to RL under continual task changes, and suggest that training under perturbations in weight space may be a useful mechanism for continual learning more broadly.
Pick a starting point or write your own. Challenges run in the background, so you can keep reading while the AI investigates.
No challenges yet. Disagree with the review? Ask the AI to revisit a specific claim.