On the Convergence of Semi-Gradient TD: A Mean Field Perspective

The paper's central mechanism---decay of the frozen-target optimality gap $G(t)$ combined with contraction of the Gibbs response map $T$---is compelling and cleanly executed. The authors write: 'If $\rho\tau>L_p$, then the frozen-target optimality gap $G(t):=F(q_t,q_t)-F(q_t,T(q_t))$ decays exponentially as $G(t)\leq e^{-2(\rho\tau-L_p)t}G(0)$.' They further show that, under the same condition, 'the map $T$ is $L_p/(\rho\tau)$-contractive in Wasserstein distance and therefore admits a unique fixed point $q_\star=T(q_\star)$.' This two-step structure provides genuine insight into why semi-gradient TD can be stable despite the moving target, and the extension to soft Q-learning demonstrates the portability of the argument.

The primary concern is the stringency of the stability condition. The paper acknowledges that 'the same regularization that stabilizes the learning dynamics introduces a quantitatively controlled bias in the learned value function,' yet the required $\rho\tau > L_p$ may be hard to satisfy without large entropy or quadratic regularization. Proposition 3 uses Bakry--Emery to certify the uniform log-Sobolev inequality, but the resulting sufficient condition is $\lambda>D_\Delta B_2$, where large $\lambda$ strongly confines the parameter distribution.\n\nA second concern is the gap between theory and empirical validation. The authors state: 'We do not provide finite-width, time-discretization, or stochastic-gradient guarantees in this work.' The numerical experiments on Asterix and Breakout are purely illustrative; the authors admit they 'do not estimate $\rho$ or $L_p$ in these environments, so they are not a direct verification of the sufficient condition $\rho\tau>L_p$.' Without finite-particle or step-size analysis, it is unclear whether the population mean-field limit has practical predictive power.

The evidence supports the formal claims inside the mean-field model, and the paper is careful to qualify its scope. Table 1 fairly contrasts the guarantees with prior results: the authors write, 'Previous results establish either feature learning or last-iterate convergence, but not both simultaneously.' Theorem 2 honestly separates the error into an approximation term and a regularization term, giving $\|V_{\nu_\star}-V^\pi\|_{L^2(d^\pi)}\leq\inf_\nu\left\{\frac{1+\gamma}{1-\gamma}\|V_\nu-V^\pi\|_{L^2(d^\pi)}+\sqrt{\frac{\tau}{1-\gamma}D_{\mathrm{KL}}(\nu\|g_{\lambda,\tau})}\right\}$. One could ask whether the comparison fully accounts for the fact that Cai et al. (2019) and Cayci et al. (2023) give finite-time guarantees in different (overparameterized but non-mean-field) regimes, but the authors do note these are 'complementary' rather than directly comparable.

As a theory paper, reproducibility hinges on proof detail rather than code release. The appendices (A.1--A.2) do supply explicit constants for the two-layer network parameterization, including the mixed-smoothness constant $L_p=\gamma\mathbb{E}_{s,a,s'}[L_\phi(s)L_\phi(s')]$ and the Bakry--Emery coefficient $\rho_{\mathrm{BE}}=(\lambda-D_\Delta B_2)/\tau$. However, no software is mentioned, and the MinAtar experiments lack hyperparameter tables in the provided text (they are deferred to Appendix D, which is referenced but not shown). Independent reproduction of the empirical curves would therefore be difficult. More importantly, the absence of finite-width or discretization bounds means there is no quantitative recipe for translating the idealized mean-field guarantees into a working algorithm with a realistic network width and step size.