Ground-Truth-First Benchmarking of Reinforcement Learning via Converse Optimality: From Smooth, Fully Observed Systems to Non-Smooth or Partially Observed Tasks
This paper attacks a foundational weakness in RL evaluation: standard benchmarks rank algorithms by relative return without knowing the optimal policy, so they cannot measure true distance to optimality. The authors propose converse optimality, a construction that generates control tasks together with certified optimal value functions and policies, enabling absolute metrics like optimality gap and regret. The framework separates continuous-time sensitivity studies from native discrete-time oracle-referenced comparisons, and extends the certificate logic to non-smooth values, non-affine action interfaces, and partially observed settings. By turning benchmarks into known-answer instruments, the work promises to expose failure modes and regime dependencies that relative rankings completely hide.
The paper presents a methodologically compelling case that RL benchmarking should move from relative return rankings to absolute optimality-gap measurement. The core argument---that relative benchmarks cannot distinguish good algorithms from lucky ones when the optimum is hidden---is well-supported by citations to known reproducibility crises and by the authors' own regime-dependent ranking experiments. However, the manuscript over-reaches by claiming a unified framework when the most interesting extensions (non-smooth values and partial observability) are presented as pure theory without empirical validation, and several key experiments reproduce prior work rather than testing novel contributions.
The ground-truth-first concept is genuinely valuable, and the separation between Mode I (sensitivity via continuous-time generators) and Mode II (absolute oracle-referenced comparison via native discrete-time fixtures) is principled and well-motivated. The empirical demonstration that PPO, TD3, and SAC rankings invert across the serial-random, serial-fixed, and NVDEx regimes is precisely the kind of diagnostic insight that relative benchmarks cannot provide. The mathematical extensions to non-affine interfaces via virtual-input pullback (ConverseQuad) and finite-covariance non-Gaussian disturbances are carefully derived and preserve the certificate logic exactly.
The manuscript suffers from a significant theory-experiment gap: the non-smooth converse optimality theory (Section 7) and the partially observed extension (Section 8) are mathematically elaborate but completely lack empirical validation, leaving it unclear whether they yield practical benchmarks or merely technical existence results. The non-smooth construction is furthermore restricted to deterministic continuous-time systems, whereas most RL operates in stochastic discrete-time settings. Mode I introduces a tension with the ground-truth-first branding: because the learner experiences a discretized version of a continuous-time generator, the actual MDP optimum depends on integration step $\Delta t$ and numerical scheme, meaning there is no fixed ground truth for the learner. Finally, the framework's practical coverage is limited by the need to hand-design a value function and policy before generating environments, which may bias the resulting task distribution toward certifiable rather than naturally occurring control problems.
The evidence strongly supports the central claim that oracle-referenced metrics reveal structure invisible to relative rankings, with the regime-dependent algorithm orderings in Section 5.5 serving as convincing proof-of-concept. Comparisons to related work are fair and thorough: the authors clearly distinguish converse optimality from inverse optimal control and from standard benchmark suites. However, the authors sometimes overstate the generality of their results. For example, ConverseQuad results are reproduced entirely from Ibrahim et al. (2026) with no new runs, and the claim that the methodology extends beyond smooth affine Gaussian systems is only weakly tested because the non-smooth and POMDP cases have no accompanying experiments. The sensitivity experiment in Section 4.3 is also limited to a single design knob (Adam vs. RAdam) on a single family of unstable 3D problems.
The authors state that code, fixtures, and diagnostics are available on a public project page, and they use Common Random Numbers (CRN) schedules to ensure that metric differences reflect algorithmic variation rather than noise realizations---both strong practices. Hyperparameters and training protocols for the oracle-referenced experiments are summarized in Appendix G, though the main text does not always surface them clearly. A barrier to independent reproduction is that ConverseQuad results are reproduced from prior work rather than newly generated for this manuscript, meaning a reproduction effort must locate and reconcile artifacts from two papers. The non-smooth and partially observed sections contain no experiments at all, so reproducibility concerns there are moot: there is nothing to reproduce beyond the analytical derivations.
Benchmarking in reinforcement learning is usually relative: algorithms are ranked by returns on environments with unknown optima. Distance to optimality remains unobservable, while conclusions may depend on reward design, stochasticity, hyperparameters, environment selection, and simulator details. We develop a ground-truth-first framework based on converse optimality, constructing each task with a certified optimal value function and policy withheld from learners and used only for evaluation. The framework unifies two modes: sensitivity benchmarking samples certified environments from continuous-time generators to assess robustness to task variation, while oracle-referenced comparison uses native discrete-time generators to evaluate policies via optimality gap, regret, and policy error. Continuous-time models support physically interpretable sensitivity studies but introduce dependence on the simulator step ($\Delta t$) and numerical scheme. Native discrete-time models align the environment, oracle, and learner on the same transition law, eliminating external integration choices as a confounder. We reproduce quantitative results covering optimizer sensitivity and oracle-referenced comparisons on serial planar arms, dynamically extended nonholonomic vehicles, and quadrotor-inspired tasks. The methodology extends beyond smooth affine Gaussian systems: virtual-input pullback handles non-affine raw-action interfaces and finite-covariance non-Gaussian disturbances. A non-smooth converse optimality formulation enables certified benchmarks with kinked value functions and switching surfaces. For partially observed evaluation, an information-state formulation reduces benchmarking to converse optimality on a sufficient statistic and provides an information-penalty reference. Code, fixtures, and diagnostics are available on the public project page.
Pick a starting point or write your own. Challenges run in the background, so you can keep reading while the AI investigates.
No challenges yet. Disagree with the review? Ask the AI to revisit a specific claim.