RM-OPD: On-Policy Distillation for Generative Reward Models
RM-OPD targets a real bottleneck in generalist reward modeling: training on mixed-domain data with standard RL produces cross-domain interference and sparse outcome rewards, so capabilities like math verification, code review, and safety guardrails cannot coexist without degrading each other. The core idea is a two-stage framework—first cultivate domain-specialist reward teachers via single-domain GRPO, then distill them into one student through routed on-policy distillation, domain contribution reweighting, and criteria context distillation. This matters because it swaps coarse verdict-level supervision for dense token-level guidance, raising the multi-domain performance ceiling for generative judges.
RM-OPD presents a well-motivated and empirically solid solution to the 'seesaw effect' in multi-domain GRM training. The shift from sparse joint RL to routed dense distillation is conceptually clean and produces consistent gains on 7B-scale models across math, code, and safety benchmarks. However, its claims of an emergent teacher-surpassing effect should be read cautiously given the limited model scale, mixed out-of-domain results, and the fact that the distilled student can also inherit and amplify teacher errors.
The diagnosis of Mix-RL's limitations is rigorous: the paper derives cross-domain gradient interference analytically and corroborates it with a clear seesaw experiment. The three proposed components are each ablated, and the main results are robust across two base model families and eight benchmark suites. Most convincingly, RM-OPD preserves strong domain-specific performance as new domains are added, and it even surpasses isolated specialists on some held-out tasks.
Several issues temper enthusiasm. First, criteria context distillation lengthens judgments substantially—from 3,488 to 4,395 tokens—raising inference costs without an accuracy-per-token control. Second, out-of-domain generalization is mixed: while some unseen domains improve, on RMB pairwise creative writing RM-OPD underperforms Mix-RL (64.96% vs 67.72%). Third, DCR target ratios are set heuristically to 1:1:1 with no sensitivity ablation, and the entire study is limited to 7B models.
These limitations are partly acknowledged, but the 'teacher-surpassing' framing remains optimistic given the narrow scale and the fact that the student inherits teacher biases without a mechanism to correct them.
The evidence broadly supports the claim that dense on-policy distillation improves over sparse Mix-RL for multi-domain GRMs. However, the novelty relative to concurrent multi-teacher OPD work is incremental: Ma et al. already proposed routed, multi-teacher on-policy distillation for general LLMs, and RM-OPD transfers that recipe to reward modeling. The comparison baselines are largely internal (MixRL-GRPO and domain specialists), so readers must trust the paper's implementation of the primary competitor. The 'teacher-surpassing' narrative is also partially a selection effect, as the student is evaluated on breadth-oriented benchmarks while specialists are narrow.
Hyperparameters, training data sources, and prompt templates are documented in Appendices B and D, but no code, checkpoints, or teacher model weights are released. Reproduction requires per-domain GRPO teacher training followed by multi-node distillation with FSDP, vLLM, and custom log-probability routing; the infrastructure barrier is significant. Additionally, because full-vocabulary teacher logits are needed, closed-source judges cannot be used, narrowing applicability.
Reinforcement Learning on Mixed-domain preference data (Mix-RL) has become the mainstream approach for training general-purpose Generative Reward Models (GRMs). However, Mix-RL suffers from two critical bottlenecks: the reward sparsity induced by outcome rewards, and the cross-domain interference arising from jointly optimizing heterogeneous capabilities, which together give rise to imbalanced capability acquisition and pervasive reward hacking. Inspired by the success of On-Policy Distillation (OPD) in multi-domain capability integration, we introduce RM-OPD (Reward Modeling via On-Policy Distillation), a two-stage framework that separates specialist acquisition from capability integration. First, we cultivate domain-specialized teacher models via single-domain GRPO fine-tuning, allowing each expert to reach its performance ceiling in isolation; then we consolidate these teachers into one student model through (i) routed on-policy distillation, which provides dense signed token-level supervision on student-generated critiques; (ii) domain contribution reweighting, which reweights token guidance toward target domain shares, preventing specific domains from dominating optimization; and (iii) optional criteria context distillation, which conditions teachers on domain-specific evaluation criteria to sharpen token-level supervision. Across multiple RM benchmarks and two base models, RM-OPD consistently outperforms vanilla Mix-RL and other strong baselines, achieving overall superior performance and exhibiting an emergent "teacher-surpassing" effect. These results establish RM-OPD as a promising paradigm for building generalist GRMs.
Pick a starting point or write your own. Challenges run in the background, so you can keep reading while the AI investigates.
No challenges yet. Disagree with the review? Ask the AI to revisit a specific claim.