PROMO: Preference-conditioned Multi-Objective Reinforcement Learning for Quadrupedal Robots
Quadrupedal locomotion requires balancing command tracking, stability, and energy efficiency, yet conventional RL bakes these trade-offs into a static scalar reward that cannot change after training. PROMO proposes a preference-conditioned multi-objective framework that takes the relative priority vector $w_t \in \Delta^{K-1}$ with $K=3$ as a runtime input to a single policy, while keeping embodiment-specific locomotion priors fixed. On a Unitree Go2, the same policy transferred zero-shot modulates behavior through preference changes alone, cutting specific energy by up to $30.4\%$, position error by $38.7\%$, and peak body-attitude deviation by $59.0\%$.
PROMO is a well-motivated and practically compelling contribution. By separating semantic operator objectives from fixed locomotion priors, the paper makes a strong case for preference-conditioned MORL as a deployable robot interface rather than merely an offline Pareto optimizer. Simulation evidence is robust, but hardware validation is limited to four anchor preferences with five trials each, and linear scalarization restricts coverage to supported (convex) Pareto regions. The authors candidly note these limits, yet the core claim of a zero-shot semantic interface on hardware is demonstrated clearly enough to advance the state of the art.
The semantic objective/prior factorization is the central conceptual advance and is empirically justified: ablations show the fixed-prior design achieves a $6\%$ fall rate against $21.8\%$ for alternatives that expose prior terms to preference conditioning. The dense $100$-preference evaluation—with $67$ non-dominated solutions out of $100$—and the action-diversity regularizer demonstrate that the policy forms a broad, responsive trade-off surface rather than interpolating a few specialist modes. Zero-shot transfer to a physical Unitree Go2 with measured modulation of energy, tracking error, and attitude deviation validates the deployment premise.
Hardware evidence is suggestive but statistically thin: only five trials per anchor and no reported confidence intervals or significance tests, as the authors concede that the study demonstrates modulation rather than dense controllability. The preference–objective correlation is measured by Spearman rank ($\rho_i$), which checks monotonicity but does not guarantee predictable interpolation between arbitrary preferences; indeed, the stability preference in the slow regime yields a velocity RMSE of $2.203$~m/s, showing that the objectives are coupled in non-obvious ways. Additionally, the 'semantic' labeling is a manual design choice: the fixed priors encode implicit value judgments (contact timing, posture) hidden from the operator. Finally, the reliance on linear scalarization means unsupported non-convex Pareto regions are not represented, a limitation acknowledged but not quantified.
The simulation comparisons are methodologically sound and well-controlled: PROMO is evaluated against shared-scale locomotion baselines (RMA, TAR), a preference-conditioned MORL baseline (MOPPO), a multi-policy Pareto method (DPMORL), and independent scalar specialists under identical scripted commands and disturbances. PROMO records higher mean scalarized return ($3.964$) and a lower fall rate ($8.4\%$) than MOPPO, and comparable efficiency to DPMORL with far fewer failures. However, comparing a single-policy method to a multi-policy front approximation like DPMORL on per-policy fall rate conflates algorithmic differences with deployment convenience. Moreover, while PROMO beats the specialists on tracking and efficiency at their respective anchors, the stability specialist still outperforms it on stability ($-1.916$ vs. $-2.245$), indicating that amortizing all trade-offs into one policy incurs small but real specialization penalties.
Reproducibility is strong on the software side: the authors pledge open-source release of the complete experimental pipeline, including baselines and evaluation protocols. Appendix B provides substantial network, simulator, and optimizer details, and the $4096$-environment parallel training configuration is specified. The main barriers are computational (large-scale GPU training in Isaac Sim/IsaacLab) and hardware-specific (Unitree Go2 with Vicon and a $200$ Hz low-level control stack); the available text truncates Appendix B before listing all final hyperparameters. Exact IsaacLab version dependencies and random seeds are not discussed in the main text, which could challenge bit-wise reproduction.
Quadrupedal locomotion requires balancing conflicting objectives such as command tracking, stability, and energy efficiency, yet conventional reinforcement learning (RL) hardcodes these priorities into a fixed scalar reward at training time. We present PROMO (Preference-Conditioned Multi-Objective Reinforcement Learning), a semantic multi-objective approach that makes this trade-off an explicit runtime input to a single locomotion policy. PROMO conditions the policy on deployment facing preferences while keeping embodiment-specific locomotion priors fixed, thereby separating operator intent from reward shaping terms required for viable gait generation. Compared with fixed-objective controllers, multi-objective baselines, and independently trained specialists, PROMO achieves objective specialization and robustness from a single deployable policy. Across 100 sampled preferences in simulation, 67 behaviors are non-dominated under exact Pareto dominance, with a mean preference-objective correlation of 0.843, demonstrating broad Pareto coverage and predictable preference response. The same policy transfers zero-shot to a Unitree Go2, where preference changes alone reduce specific energy by up to 30.4%, position error by 38.7%, and peak body-attitude deviation by 59.0% relative to the balanced preference. These results establish preference-conditioned multi-objective RL as a practical runtime interface for adaptive legged locomotion, extending its role beyond offline Pareto-set construction. Open-source code and videos are available at https://amrmousa.com/promo/.
Pick a starting point or write your own. Challenges run in the background, so you can keep reading while the AI investigates.
No challenges yet. Disagree with the review? Ask the AI to revisit a specific claim.