Selective Growing Q-Networks (SGQN): An Analysis of Adaptive Action Grids in Decoupled Q-Learning
SGQN introduces selective growth and lazy pruning into decoupled Q-learning for continuous control, aiming to concentrate action-grid resolution where the value function peaks. On eight DMControl tasks it reports competitive mean returns, but its central—and most valuable—finding is that the adaptive grid is not responsible for the performance: growth behaves like uniform expansion, pruning changes returns by at most 1.1%, and what actually matters is starting from a coarse grid and using $\varepsilon$-greedy exploration. The paper is therefore best read as a rigorous self-critical analysis of which ingredients in growing action-grid methods matter and which do not.
The paper is refreshingly self-critical and methodologically sound in its internal analysis, but its comparative claims against GQN and DecQN are confounded by multiple unablated differences. The authors are transparent about this, writing "we do not attribute its advantage to a single component" and noting that "growth is effectively uniform" in practice. The work should be read as a valuable ablation study and a competitive baseline release rather than evidence that value-guided discretisation improves decoupled Q-learning.
The internal ablations are rigorous and clearly presented. The coarse-start ablation shows that initialising with all 9 bins produces "individual seeds that collapse" on some tasks, while growing from 3 anchors prevents this. The $\varepsilon$-greedy ablation is devastating: "annealed softmax lowers the return by 23% to 100%" and on Hopper Hop it reaches 0. The direct measurement of Q-value spreads across bins—showing a median range of only 0.5%–2.9% of the mean—is compelling evidence that the growth threshold $\tau_Q$ cannot discriminate between bins.
The comparison to GQN and DecQN is heavily confounded. Table 1 reveals SGQN uses one gradient update per environment step versus one per eight for GQN and DecQN, a difference the authors describe as "a candidate that we have not tested" and which alone could explain much of the return gap.
Additionally, SGQN decays $\varepsilon$ from 0.1 to 0.01 while baselines keep it constant at 0.1, uses a different network (layer-norm MLP versus residual layer-norm), and uses $\beta=0.4$ versus 0.2. Since the adaptive grid itself does not affect performance, these confounds become the primary explanation for the results, yet none are ablated against the baselines. The hyperparameter sensitivity study is also underpowered, using only 3 seeds on 2 tasks, which makes it impossible to reliably distinguish signal from noise in the reported variations.
The evidence strongly supports the negative internal claims—that selective growth and pruning are inert—but weakly supports the positive comparative claim that SGQN beats GQN. The authors report that "removing pruning changes the return by at most 1.1%", yet they still claim SGQN outperforms GQN by large margins on six of eight tasks without controlling for update frequency, network architecture, or exploration schedules.
They correctly concede that "we did not run GQN with the update frequency or other settings of SGQN," which is a major methodological gap: without this control, the comparison is apples-to-oranges. The Bang-Bang MPO baseline uses a fundamentally different algorithm class, and CQN learns from pixels while all others use states, further weakening the comparative framework.
The authors provide detailed hyperparameter tables and algorithm pseudocode, which is commendable. However, reproducibility is hampered by baseline runs that "produced no data" (e.g., all seven Bang-Bang PPO runs on Reacher Hard), suggesting fragility in the evaluation code or environment setup that is not explained.
No code or data repository is mentioned in the text. The main results rely on 7 seeds, which is adequate but marginal. Crucially, because the confounded baseline differences are not crossed in a factorial design, independent researchers cannot determine which ingredient produces the reported returns, and the authors explicitly warn that "we cannot say which of these differences accounts for the advantage."
Value-based Q-learning for continuous control can discretise the action space, but the resolution of the discretisation trades early exploration against final precision. Growing Q-Networks (GQN) grow the resolution uniformly from coarse to fine. We introduce Selective Growing Q-Networks (SGQN), which keeps decoupled Q-learning and replaces uniform growth with an adaptive action grid: bins are added next to bins with high estimated Q-value, bins with low Q-value are pruned, and actions are chosen epsilon-greedily. On eight DeepMind Control Suite (DMControl) tasks with 2 to 12 action dimensions, SGQN has the highest mean final return among the compared methods on four, is within 1.1% of the best method on three more, and is 3.4% below Bang-Bang MPO on Walker Run, using on average 7.5 of 9 available bins per dimension. We then test whether the adaptive grid explains these results, and find that it does not. Growth activates every candidate bin and behaves like uniform growth, and removing pruning changes the return by at most 1.1%. Two other choices matter: starting from a coarse grid, since starting with all 9 bins produces collapsed seeds, and epsilon-greedy selection, since an annealed softmax lowers the return by 23% to 100%. SGQN is insensitive to its growth and pruning hyperparameters. It also differs from the GQN and DecQN baselines in other respects, notably one gradient update per environment step instead of one per eight, which we did not control for, so we do not attribute its advantage to a single component. We present SGQN as a competitive value-based method and as an analysis of which ingredients of a growing action grid matter.
Pick a starting point or write your own. Challenges run in the background, so you can keep reading while the AI investigates.
No challenges yet. Disagree with the review? Ask the AI to revisit a specific claim.