Optimal Transport Meets Reinforcement Learning: A Survey
This survey systematises the sprawling literature at the intersection of optimal transport and reinforcement learning through a four-question taxonomy: what role OT plays, which distributions are compared, which formulation is used, and how temporal structure is handled. By foregrounding the geometry-aware nature of transport-based discrepancies, the authors explain why OT can outperform $f$-divergences when learner and expert distributions overlap weakly—a common scenario in imitation learning, offline RL and robust control. The result is a broad methodological map spanning occupancy matching, trust regions, world-model learning and multi-agent settings.
The paper delivers a comprehensive and well-organised panoramic view of OT inside RL algorithms. Its recurring four-question framework successfully unifies literatures that are usually treated in isolation, from adversarial imitation to Wasserstein trust regions and robust control. The authors are careful to note scope limits—excluding purely application studies and avoiding empirical horse races—making the work a reliable methodological reference, though not a practical guide for practitioners seeking off-the-shelf algorithmic recommendations.
The four-criterion taxonomy (role, distributions, formulation, temporal structure) is the strongest analytical contribution, turning a sprawling literature into a navigable map. The preliminaries are mathematically rigorous yet signposted with intuitive explanations—such as the occupancy-measure flow constraint and the sand-pile intuition for OT—making the survey useful for readers who are not deep experts in both fields. The breadth is genuinely broad, with substantive sections on imitation, policy optimisation, model-based and offline RL, robustness, safety and multi-agent settings, all connected back to the same organising questions.
The survey deliberately avoids empirical performance comparisons and focuses almost exclusively on methodological developments.
While the authors justify this choice by noting environment dependence, the result is that readers receive little guidance on which OT variant actually works best in practice, or on the computational cost-accuracy trade-offs of different Sinkhorn formulations across benchmarks. Additionally, the notation is dense—over 190 bibliographic entries and dozens of mathematical objects are introduced in the first twenty pages—which may hinder the accessibility the authors claim for readers with only a basic mathematical background.
The evidence presented is largely bibliographic and conceptual, which is appropriate for a survey, but it means that comparative claims about OT's superiority over $f$-divergences rest on citations to individual papers rather than on a unifying theoretical analysis conducted here. The comparisons to related survey efforts are fair and specific: the authors correctly note that Montesuma et al. (2024) treats RL only briefly and that Baheri and Kochenderfer (2024) limits itself to multi-agent settings at a high level. These distinctions are substantiated by the far broader coverage on offer here.
As a survey, the paper does not introduce new experiments, code or datasets, so standard reproducibility criteria do not apply in the usual sense. However, the paper is highly reproducible as a reading guide: methodological details for each cited method are extracted and presented with consistent notation, and the authors explicitly state inclusion criteria and search strategy. What would block independent verification is the lack of a persistent code repository or supplementary database of the surveyed papers; practitioners hoping to reproduce any specific algorithm must still consult the original sources, whose hyperparameters are only discussed conceptually here.
Reinforcement learning (RL) algorithms frequently compare probability distributions, such as state visitation distributions induced by policies and experts, action distributions from learned policies and offline datasets, or transition distributions from learned models and environments. However, commonly used divergences may become ineffective when these distributions overlap weakly, which is frequently encountered in imitation learning, offline RL, and deployment under distribution shift. Optimal transport (OT) offers an alternative by measuring the cost of \emph{moving} probability mass from one distribution to another under a ground cost that encodes task geometry. This survey covers how OT is used inside RL objectives and algorithms. For each method, we identify: the role OT plays, the distributions compared, the OT formulation used, and the treatment of temporal structure. Beyond categorising existing methods, we discuss the motivations behind different OT choices, practical considerations such as cost design and computational challenges, and highlight open problems including scalable trajectory-level transport, principled handling of mass mismatch, and theoretical analysis for OT-regularised RL.
Pick a starting point or write your own. Challenges run in the background, so you can keep reading while the AI investigates.
No challenges yet. Disagree with the review? Ask the AI to revisit a specific claim.