Reachability-Informed Reinforcement Learning for Multi-Impulse Interplanetary Transfers
This paper tackles deterministic multi-impulse interplanetary rendezvous by embedding classical astrodynamics inside a reinforcement-learning policy. The proposed Reachability Analysis-Informed RL (RARL) framework maps bounded velocity perturbations into a local ellipsoidal waypoint set via the first-order position-from-velocity sensitivity matrix; the policy selects a waypoint, and a Lambert solver reconstructs the dynamically consistent impulsive arc. The core contribution is a physically structured action interface that supports both near-optimal nominal transfers and policy reuse across a family of dispersed departure states.
RARL is a carefully constructed hybrid of geometric reachability and deep RL that yields strong empirical performance on an Earth–Mars benchmark. Across three independent seeds, the mean maneuver cost is 10.23 km/s, only 1.72% above a validated sequential convex programming reference, and multi-state training attains 100% feasibility on 10,000 held-out Monte Carlo departures with a mere 0.61% increase in nominal cost. The limitation is that feasibility is purely empirical: as the authors note, membership in the linear set does not guarantee that the Lambert-reconstructed impulse obeys the same limit, and the study is confined to fixed-node two-body dynamics.
The reachability-informed action interface is the standout element. It replaces generic continuous actions with waypoint selection inside an STM-derived ellipsoid, making policy decisions geometrically interpretable. The supporting diagnostics are equally convincing: the local linear map achieves a mean relative error of $9.52 \times 10^{-5}$ at maximum impulse (Fig. 3), while the terminal two-impulse proxy ranks nonlinear Lambert demand with Spearman $\rho_s = 0.9991$ (Fig. 4). The multi-state results are robust across seeds, with every trained policy achieving zero cap violations on dispersed departures compared with a 6.49% mean feasibility rate for single-state policies (Table 6).
The local reachability description is an approximation without certification. The authors explicitly state that The first-order set is therefore used as a local decision interface rather than as a nonlinear reachability or recursive-feasibility certificate, meaning impulse-limit compliance is verified only a posteriori via Lambert reconstruction. This empirical safety model is mitigated by a small inner guard $g$, but no analytic margin is provided.
A second major restriction is the reliance on deterministic two-body heliocentric dynamics throughout propagation and reconstruction; perturbations, ephemeris, and real-world operational constraints are absent. Finally, the node count and segment durations are fixed ($N=20$), so the policy learns where to steer within a predetermined mesh but not how to place maneuvers optimally in time.
The evidence supports the central claims for the studied benchmark. The SCP reference is independently propagated to verify consistency, producing a 10.0585 km/s baseline against which RARL’s 1.72% gap is meaningful. The comparison to Marmo et al.’s reported 10.4048 km/s deterministic result is used only to situate RARL within the literature, while the primary comparison to the authors’ own SCP reference is fair and internally consistent. The single-state versus multi-state evaluation on a common 10,000-departure Monte Carlo set directly demonstrates the robustness–cost trade-off, and the paper commendably reports both per-seed and aggregate statistics.
Experimental details are unusually thorough: nondimensionalization, observation scaling bounds, reward weights, MLP architectures, and PPO hyperparameters are all tabulated (Tables 1–3), and the software stack—PyKEP, Stable-Baselines3, PyTorch, Gymnasium, SciPy, CVXPY, MOSEK—is mainstream and well documented. Reproduction is nonetheless hindered by enormous sample budgets: Each single-state run uses approximately 240 million training transitions. No code repository or random seeds are mentioned, and the multi-step checkpoint-selection heuristic adds procedural complexity that would need to be duplicated exactly.
Reinforcement learning offers the prospect of a reusable sequential decision-making mechanism for spacecraft trajectory design, motivating policy interfaces that connect learned decisions to the underlying maneuver geometry. This paper develops Reachability Analysis-Informed Reinforcement Learning (RARL) for deterministic multi-impulse interplanetary transfers, placing intermediate waypoint selection at the center of the learned decision process. Local first-order reachability maps bounded velocity perturbations into an ellipsoidal set of next-node positions, within which the policy selects its waypoint. Lambert reconstruction then determines the corresponding maneuver to reach this selected waypoint along a dynamically consistent ballistic arc, coupling learned transfer-geometry selection with classical astrodynamics. A terminal two-impulse reconstruction completes the rendezvous, supported by a linear maneuver-demand assessment used for reward shaping. Numerical studies characterize this interface on a two-body Earth-Mars benchmark. Across three independent training runs, RARL achieves a mean maneuver cost of 10.23 km/s, 1.72% above a validated local sequential convex programming reference. Training over dispersed initial states extends policy reuse across a departure family with fixed target state and transfer duration. Each of the three independently trained multi-state policies completes all 10,000 held-out Monte Carlo departures without impulse-cap violations, compared with a mean feasibility rate of 6.49% for single-state policies. This broader sampled feasibility is accompanied by a 0.61% increase in mean nominal maneuver cost, without further training across departures. These results demonstrate that a reachability-informed decision interface supports benchmark-quality trajectory construction and policy reuse across dispersed departure conditions.
Pick a starting point or write your own. Challenges run in the background, so you can keep reading while the AI investigates.
No challenges yet. Disagree with the review? Ask the AI to revisit a specific claim.