CurvZO: Adaptive Curvature-Guided Sparse Zeroth-Order Optimization for Efficient LLM Fine-Tuning
CurvZO tackles the memory wall problem in LLM fine-tuning by proposing a zeroth-order optimization method that tracks curvature signals online from scalar feedback instead of requiring pre-computed statistics. The core idea uses curvature-aware importance sampling to select which parameters to perturb in sparse ZO updates, coupled with an adaptive budget mechanism that adjusts sparsity based on the evolving curvature distribution. This matters because existing sparse ZO methods either rely on costly pre-computed Fisher information or use static/random sparsity patterns that may be suboptimal.
CurvZO presents a technically sound incremental contribution to sparse zeroth-order optimization for LLM fine-tuning. The paper's key innovation—tracking curvature online via $s_i \triangleq \Delta^2 v_i^2$ and using it to construct a variance-minimizing sampling distribution $\pi_i \propto \sqrt{S_i}$—is well-motivated and theoretically grounded. The Horvitz-Thompson bias correction (Definition 3.1) and the adaptive budget formulation based on effective support size $d_{\text{eff}}$ and sharpness $H$ (Eq. 20) demonstrate careful design. However, the work is more incremental than transformative: the convergence rate of $\mathcal{O}(1/T)$ matches standard ZO guarantees, and the claimed improvements over baselines are modest (up to 4.4 accuracy points, 2× speedup). The block-wise approximation (Section 3.5), while necessary for scalability, does sacrifice the per-parameter granularity that differentiates CurvZO from layer-wise methods like DiZO.
The theoretical framework is solid: Proposition 3.2 correctly establishes unbiasedness of the Horvitz-Thompson corrected estimator up to $\mathcal{O}(\epsilon^2)$, and the variance bound $\text{Var}(\tilde{g}_i^t) \leq C F_{ii}/\pi_i^t$ (Proposition 3.3) leads naturally to the optimal sampling rule $\pi_i \propto \sqrt{F_{ii}}$ (Proposition 3.4). The choice to use the biased but low-variance curvature score rather than its unbiased but high-variance variant (Appendix B) is pragmatically justified for LLM-scale optimization. The adaptive budget mechanism elegantly combines effective support size and entropy to modulate exploration, and the block-wise extension (Propositions 3.9–3.10) preserves estimator properties while controlling overhead.
First, the curvature score expectation (Eq. 6) reveals a fundamental limitation: $\mathbb{E}[s_i] = \underbrace{\pi_i \sum_j \pi_j g_j^2}_{\text{shared term}} + \underbrace{\pi_i(3-\pi_i)g_i^2}_{\text{signal term}} + \mathcal{O}(\epsilon^2)$. The shared term couples all coordinates through $\pi_i$, meaning the curvature score tracks not just local sensitivity but also spurious global correlations. The authors acknowledge this bias but dismiss the unbiased variant due to variance concerns. Second, Theorem 3.8 assumes a fixed sampling budget, yet the method adaptively changes $B$ at each iteration; the interaction between adaptive sampling and convergence is not rigorously characterized. Third, the claim that perturbing small-magnitude parameters causes 'disproportionately large changes' in Sparse-MeZO is partially contradicted by Liu et al.'s finding that small weights are more noise-resilient. Finally, the curvature score $\tilde{s}_i = \frac{v_i^2}{\sum_j v_j^2}\Delta^2$ (Eq. 7) can misidentify important parameters when perturbation magnitude $\|v\|^2$ varies significantly across iterations.
The empirical evaluation primarily compares CurvZO against MeZO and DiZO on SuperGLUE classification tasks (SST-2, RTE, CB, BoolQ, WSC, WIC) and generation tasks (SQuAD, DROP) for OPT and Llama2 models. Results in Table 1 and Figure 2 show consistent improvements: on OPT-2.7B, CurvZO achieves 66.8% average accuracy versus MeZO's 64.2%, and reaches convergence in ~2× fewer iterations on BoolQ and RTE (Figure 3). However, the comparison scope is narrow: SensZOQ (using pre-computed Fisher) and HiZOO (Hessian-informed ZO) are cited but not directly compared, despite HiZOO reporting 8× speedups on similar tasks (Zhao et al.). The comparison against DiZO is particularly unfavorable—Table 1 shows DiZO actually underperforms MeZO on most tasks, suggesting it may not be the strongest baseline. The memory overhead claim (~0.01 GB increase on OPT-6.7B, Table 3) is credible given block-wise scores, but the speedup attribution lacks ablation: is the gain from adaptive budget, curvature sampling, or both?
Reproducibility is moderately strong. The paper provides algorithmic details for the curvature score computation, EMA smoothing, and adaptive budget selection, with hyperparameters $B_{\text{min}}, B_{\text{max}}, \alpha, \beta$ specified in Appendix I. Code is released at an anonymous repository. However, critical implementation details are underspecified: (1) the exact block granularity (tensors vs. layers) and how blocks are partitioned for different model architectures; (2) the threshold for clipping $\pi_i^{t\star} \leq 1$ is mentioned but its impact not studied; (3) numerical stability handling when $S_i^t \approx 0$ for coordinates with negligible curvature; (4) initialization of curvature scores before any observations. The perturbation scale $\epsilon$ and learning rate $\eta$ are not reported in the main text. Independent reproduction would benefit from explicit pseudocode for the full algorithm including adaptive budget updates.
Fine-tuning large language models (LLMs) with backpropagation achieves high performance but incurs substantial memory overhead, limiting scalability on resource-constrained hardware. Zeroth-order (ZO) optimization provides a memory-efficient alternative by relying solely on forward passes, yet it typically suffers from slow or unstable convergence due to high-variance gradient estimates. Sparse ZO updates partially address this issue by perturbing only a subset of parameters, but their effectiveness hinges on selecting informative parameters, which is challenging in ZO optimization because each query yields only scalar feedback. We propose \textbf{Adaptive Curvature-Guided Sparse Zeroth-Order Optimization (CurvZO)}, which tracks curvature signals online from scalar ZO feedback and leverages these signals to construct a parameter-wise sampling distribution for selecting coordinates at each update, reducing the variance of the sparse ZO gradient estimator. Moreover, CurvZO dynamically adapts the perturbation budget to the evolving curvature signal distribution, yielding sparse ZO updates that remain both focused and sufficiently exploratory. Extensive experiments on OPT and Llama across diverse NLP tasks show that CurvZO consistently improves fine-tuning performance and reduces training time over ZO baselines. It improves accuracy by up to 4.4 points and achieves up to a $2\times$ speedup, while preserving memory efficiency.
Pick a starting point or write your own. Challenges run in the background, so you can keep reading while the AI investigates.
No challenges yet. Disagree with the review? Ask the AI to revisit a specific claim.