Scaling Laws for Generative Reward Models
Generative reward models judge text by generating verdicts instead of scalars, but it remains unknown whether stronger static judges give better online training signals. This paper sweeps Qwen3 judges from 0.6B to 14B parameters, comparing thinking GRPO judges against answer-only SFT judges as rewarders for online DPO in creative writing. The central finding is an evaluator–rewarder gap: thinking models lead on fixed-distribution Gold accuracy yet usually lose on downstream policy Elo, with larger judges helping even beyond policy size. The work is framed as practical guidance rather than an isolated causal claim.
The paper demonstrates that under its fixed learned 32B Gold evaluator, thinking GenRMs trained with GRPO achieve higher static accuracy yet usually lose to answer-only SFT judges when used as online DPO rewarders (7 of 8 matched settings). This evaluator–rewarder gap is an important empirical observation, but the authors rightfully stress that 'the main result is conditional on the specified learned Gold reference; neither format neutrality nor universal writing-quality superiority is established.'
The strongest contribution is the systematic sweep across judge format, judge size, policy size, and training duration, which yields actionable guidance: larger judges help even above policy size, and intermediate checkpoints can outperform final ones as rewarders. A descriptive log-linear fit reaches an in-sample $R^{2}$ of 95.2% across the 16 matched configurations, underscoring smooth capacity effects within the grid.
The central manipulation confounds training objective and response format: the paper compares SFT answer-only judges against GRPO thinking judges, so the observed reversal cannot be attributed solely to reasoning traces. The authors note this explicitly: 'This compares two practical training pipelines, rather than isolating the causal effect of reasoning under identical objectives.'
Additionally, the learned Gold evaluator is itself a 32B model fine-tuned on implicit Reddit vote preferences, not human judgments of generated text, and validation sets are reused across Gold, proxy, and policy evaluation. A follow-up audit with a different learned judge was inconclusive, as the paper states: 'A limited follow-up using a different learned judge is inconclusive overall.'
The internal evidence strongly supports the core claim that static accuracy does not predict online rewarder quality, with clear trajectory plots and matched Elo arenas. The authors also ground their work in the broader reward-overoptimization literature. Gao et al. (2022) showed that 'optimizing too much against such a model eventually hinders the true objective, a phenomenon we refer to as overoptimization,' and Rafailov et al. (2024) found that 'DAAs exhibit a type of reward over-optimization consistent with that previously observed in RLHF.' The paper does not overstate novelty: it acknowledges that the generic evaluator–rewarder distinction has been raised before, and it limits its conclusions to the studied Qwen3 creative-writing pipeline.
The authors released code and analysis scripts, but critical per-run details are missing. They state that 'resolved per-run configurations have not been recovered for every plotted trajectory, so those original caps should be read as nominal settings rather than a newly verified run-wide invariant,' and that 'the released analysis identifies its numerical inputs separately from the historical training recipes.' Full training replay also requires model and dataset access that is not fully bundled. Consequently, an independent reproduction would be blocked by unresolved hyperparameters and incomplete configuration provenance.
We study generative reward models (GenRMs) as online preference providers for policy optimization in creative writing. Across Qwen3 reward-model sizes from 0.6B to 14B, Thinking judges trained with GRPO have higher static Gold-label accuracy than Answer-Only judges trained with SFT. This ranking often reverses when the judges train policies through online DPO: Answer-Only rewarders yield higher peak and top-five-average policy Elo in seven of eight matched settings under a fixed, learned 32B Gold evaluator. Intermediate Thinking-judge checkpoints can also outperform later, more statically accurate checkpoints as rewarders. Sweeps over rewarder size, policy size, training duration and DPO regularization show substantial benefits from reward-model capacity, including judges larger than the policy. We analyze trajectories with evaluator-specific Elo arenas, conditional match-resampling intervals and descriptive scaling fits. These results demonstrate an evaluator-rewarder gap in the studied pipelines: fixed-distribution judging accuracy does not determine downstream optimization quality. The comparison changes judge-training objective as well as response format, and the main ranking is conditional on its learned reference rather than a universal human-quality result. A limited follow-up using a different learned judge is inconclusive overall.
Pick a starting point or write your own. Challenges run in the background, so you can keep reading while the AI investigates.
No challenges yet. Disagree with the review? Ask the AI to revisit a specific claim.