Value-Guided Repulsion for Offline Reinforcement Learning
Offline reinforcement learning from static datasets demands expressive policies that remain within data support while improving value. Drifting models train a direct generator via attraction to behavior actions and repulsion among generated particles, but standard drift policies weight repulsion solely by geometric distance and therefore cannot distinguish particles by expected return. This paper proposes Value-Guided Repulsion Q-Learning (VRQL), which modulates repulsion by combining kernel distance with a normalized Q-value bias so that lower-valued neighbors exert stronger repulsive force, while preserving the behavior anchor and a separate value-maximization objective. The authors report competitive performance across 50 OGBench tasks and 6 D4RL AntMaze environments without requiring an auxiliary behavior model.
VRQL offers a clean, well-motivated modification to drifting policies: injecting estimated action values into the repulsion field is an intuitive way to shape the behavioral constraint. The empirical picture is generally positive, but the gains over the geometric predecessor are uneven across categories and the method is fundamentally exposed to critic misranking. Overall, the paper advances the drifting-policy line and provides useful ablations, though the headline claim of universal best-or-tied performance is slightly tempered by category-level variance.
The central mechanism is strongly validated. Ablations show that inverting or permuting the value biases causes active degradation, confirming that the precise alignment between low Q-values and high repulsion is essential rather than decorative. The adaptive normalization by within-state standard deviation also demonstrably stabilizes training compared with raw or absolute-mean scaling.
First, VRQL inherits an unresolved vulnerability to critic errors: if the ensemble misorders action values, the guidance mechanism will repel better actions and pull worse ones, and the paper does not present any empirical safeguard beyond ensemble min-aggregation. Second, the repulsion weights rely on stopped target-critic estimates, meaning value guidance reshapes a frozen drift target without direct gradient coupling to the value objective; the paper does not analyze whether this decoupled shape is guaranteed to align with policy improvement. Third, while gains over DriftQL are sizeable on some tasks (e.g., cube-double), they are marginal on others (humanoidmaze-large: $8 \pm 1$ versus $5 \pm 8$), and the reported tuning range for guidance temperature ($\beta \in [0.05,10]$) signals appreciable sensitivity.
The main results show that VRQL raises success rates substantially over DriftQL in several OGBench categories and exceeds the selected representatives from diffusion, flow, and Gaussian policy classes on average. However, the main table highlights only one baseline per class, and the full per-task tables in Appendix C reveal that alternatives such as Q-Flow and GFP are competitive or superior on individual tasks (e.g., puzzle-4x4 and scene-play tasks). Consequently, the evidence supports the effectiveness of value-guided repulsion but does not establish uniform dominance over all recent methods.
The paper discloses network architectures, default hyperparameters, and states that anonymized source code is provided in the supplementary materials, which is supportive of reproduction. Nevertheless, the wide per-task hyperparameter ranges ($\beta \in [0.05,10]$ and $\tau \in [0.02,0.8]$) and the lack of a documented selection protocol mean an independent reproduction would require significant re-tuning. Additionally, several core baseline citations (e.g., Deng et al., 2026; Houssaini et al., 2026) reference works dated in the future, raising immediate concerns about accessibility and independent verification of the underlying drifting model framework.
Drifting models have emerged as a highly efficient and stable paradigm for policy design in offline reinforcement learning (RL). However, standard drift policies determine repulsion solely from geometric relationships among generated actions and therefore cannot distinguish them by expected value. To address this limitation, we introduce Value-Guided Repulsion Q-Learning (VRQL) to integrate action value estimates into the particle interaction field. For each state, VRQL combines geometric kernel distance with a normalized Q value scaling. This mechanism assigns greater repulsive influence to particles with lower values while strictly retaining attraction toward behavior action. Consequently, this value-guided interaction field aligns behavior regularization with policy improvement without adding an auxiliary behavior model stage. On 50 OGBench task configurations and 6 D4RL AntMaze configurations, VRQL demonstrates highly competitive performance against established baselines, suggesting that value-guided particle interaction is an effective method for offline RL.
Pick a starting point or write your own. Challenges run in the background, so you can keep reading while the AI investigates.
No challenges yet. Disagree with the review? Ask the AI to revisit a specific claim.