Adapter Thickets: Splitting an RLVR Budget Beats Concentrating It
The paper identifies a failure mode in the standard RLVR pipeline: training one policy with reinforcement learning from verifiable rewards (RLVR) improves single-sample accuracy but can hurt majority-vote performance because RLVR increases correlation among voter errors and collapses output diversity. The proposed fix is an "adapter thicket" — splitting the same training budget across $K$ independent LoRA adapters, each trained on a disjoint random data shard. Across 16 (model, $K$) settings and four models (1.5B–8B), thickets consistently out-vote the fully trained single adapter, and for $K \geq 4$ they protect the base model's vote while keeping most of the single-sample gain.
The paper presents a clean, well-motivated idea with strong empirical support under tightly controlled conditions. The core claim — that concentrating an RLVR budget in one adapter hurts majority voting by correlating voter errors, and that splitting the budget preserves diversity — is convincingly demonstrated across multiple model families and scales. The experimental design is commendable: every method pools exactly 160 completions per problem, training uses the same total data and compute, and a step-matched control isolates the effect of training depth versus diversity. However, the scope is narrow (GRPO only, no KL penalty, $K \leq 16$, models up to 8B), and the voting analysis is limited to math because code outputs lack canonical final answers. The title's framing as "splitting beats concentrating" is largely supported, though the margin over the base model shrinks as the vote budget grows and the thicket trails the base in 7 of 16 settings at 160 votes.
The central empirical finding holds up robustly: adapter thickets beat the fully trained single adapter in all 16 (model, $K$) majority-vote settings and in all 32 (model, suite, $K$) coverage settings. The monotonic rise in mean pairwise error correlation $\bar{\rho}$ during single-adapter training, paired with the peak-and-decline pattern of majority-vote accuracy, provides a clear mechanistic explanation. The step-matched control is a strong baseline, and the finding that random shards match subject-specialist shards simplifies deployment. The paper's framing connects neatly to verified prior work on diversity collapse.
The study is limited to GRPO with KL coefficient 0.0, so it is unclear whether the observed damage is exacerbated by or independent of the absence of KL regularization — a variable known to affect diversity collapse. The voting results are math-only; code enters only through coverage, pass@1, and correlation, so the central claim about majority voting is not tested on code. The subject-versus-random ablation at $K=7$ uses a single seed, weakening its reliability. At a fixed 160-vote budget, thickets beat the base in only 9 of 16 settings, with the lead shrinking as the vote budget grows; the paper acknowledges this but the headline "beats concentrating it" slightly oversells the absolute picture.
The 2026 date on this arXiv preprint and on several citations (e.g., Gan and Isola 2026) is anomalous; while the cited papers were verifiable, readers should confirm final versions. The theoretical explanation for why random data shards produce sufficiently decorrelated errors remains thin: is it purely initialization noise and data stochasticity, or is there a deeper mechanism?
The evidence supports the main claims well within the tested regime. Comparisons are fair: total training rollouts, total data, and inference budgets are held constant across methods. The paper appropriately relates its findings to Yue et al. (2025) and Li et al. (2025b), who document diversity collapse in RLVR, and to Gan and Isola (2026), whose "Neural Thickets" concept the authors echo and extend with reward-directed training rather than random perturbation. The step-matched control correctly tests whether the thicket wins simply because no voter trains longer than a thicket member. One gap: the paper does not compare against methods that recover diversity within a single policy (e.g., DPH-RL from Li et al. 2025b), which would directly test whether splitting the budget is superior to changing the divergence objective.
The paper provides detailed hyperparameters in Appendix A (Table 4): rank-64 LoRA with targets, learning rate $2 \times 10^{-5}$, GRPO with group-normalized advantages, 8 rollouts per prompt, binary verified reward, and 1 epoch. Three seeds are used for main experiments. The data pools are specified (2,000 MATH problems, 2,000 LeetCodeDataset problems) and the evaluation benchmarks are listed. However, no code repository or data release is mentioned, and the paper relies on a specific verl-based GRPO implementation (Sheng et al., 2025) without release confirmation. Reproduction without these resources would require substantial re-implementation. The fixed-inference-budget design is clearly described but involves non-trivial canonicalization and tie-breaking rules that would need careful replication.
Majority voting over sampled completions is the workhorse of test-time scaling, and reinforcement learning with verifiable rewards (RLVR) is the workhorse for making each completion better. The standard pipeline composes the two: train one policy with RLVR, then sample it many times and vote. We show that this composition is lossy. A vote can only overturn mistakes that its voters do not share, and RLVR sharpens a policy so that its samples increasingly make the same mistakes. With every method drawing exactly $160$ completions per problem, training a single LoRA adapter on the full RLVR budget raises single-sample accuracy on every model we test ($1.5$B-$8$B). Yet on three of four models it leaves the majority vote below that of the untrained base model, by up to $4.8$ points. The damage builds during training: voter errors grow steadily more correlated, and the majority vote accuracy peaks early before falling by up to $7.0$ points. The cause is concentration, not RLVR itself. We split the same data and training budget across $K$ LoRA adapters, each trained on its own random disjoint shard, and call the result an adapter thicket. Thickets out-vote the fully trained adapter in all $16$ (model, $K$) settings, and for $K{\geq}4$ they stay within $0.8$ points of the base model or above it. A single adapter stopped early, at a thicket member's step count, is a strong control that matches thickets for small $K$. For $K{\geq}8$, thickets keep more of RLVR's single-sample gain and out-vote this control in six of eight settings. The cost of concentration also grows with the number of votes: from $16$ to $160$ votes, the thicket's lead over the fully trained adapter widens from $1.3$ to $3.3$ points. When the plan is to sample and vote, an RLVR budget is better spent broad than deep.
Pick a starting point or write your own. Challenges run in the background, so you can keep reading while the AI investigates.
No challenges yet. Disagree with the review? Ask the AI to revisit a specific claim.