What Pairwise Evaluation Misses: Reward Scores across Prompts
Reward models score responses to steer alignment and filtering, but pairwise benchmarks compare only answers to the same prompt, leaving prompt-level score shifts completely invisible. This paper shows that eighteen released scorers exhibit far larger between-prompt variance than human ratings, proposes a three-part human-referenced audit to isolate the source, and warns that blindly subtracting prompt means can actually degrade global response selection.
The paper makes a strong, practically important case that pairwise reward evaluation is insufficient for cross-prompt decisions. Its audit of eighteen scorers reveals substantial between-prompt clustering that often weakens under prompt substitution, and its matched training experiments cleanly disentangle loss objectives from supervision signals. However, the human "ground truth" itself is fragile: HelpSteer2's sequential protocol yields a near-zero between-prompt share while HelpSteer's independent ratings give 0.26, and some evaluated models trained on the validation prompts. These issues limit how firmly one can conclude that the models are wrong rather than simply differently distributed.
The proposed audit is methodically thorough, combining intraclass correlations, sibling-rating regressions, and controlled prompt substitutions with length, paraphrase, and transplant conditions. The matched training on Qwen3 at two sizes—varying regression versus Bradley–Terry objectives while holding base and data fixed—provides rare causal evidence in this space. The finding that full centering lowers helpfulness in global selection is an important practical warning.
The central human reference is unstable across datasets. The authors note that on HelpSteer2, annotators rated the two responses in sequence and "the two ratings are weakly negatively correlated, with a leave-one-out correlation of $-0.08$," producing a human between-prompt share near zero, whereas HelpSteer uses independent annotators per response and yields a share of $0.26$. Because the two sets also differ in response source, the paper admits that "we cannot separate these influences." Training-data leakage further taints some comparisons: the Skywork-Llama-8B model's preference data "contain 94 of the 511 validation first turns." The downstream selection tests are also narrow, resting almost entirely on HelpSteer, and the authors concede that the DPO filtering follow-up "establishes neither a difference nor equivalence."
A conceptual ambiguity remains in the between-prompt share metric itself. A high share can reflect either a spurious prompt-level offset or within-prompt compression of genuine quality differences; the paper notes this but offers no independent identification of which mechanism dominates for each model. The generic-prompt condition also shifts the input distribution, and 9.6% of prompts lack valid paraphrases.
The empirical comparisons are fair and well-placed against prior work. The authors correctly locate their contribution relative to pairwise debiasing methods and prompt-template offset removal. The claim that clustering does not imply human agreement is well-supported: "Replacing the original prompt with a fixed instruction keeps substantial clustering but weakens agreement with prompt-level human ratings, so clustering alone does not show agreement with humans." Global-selection evidence supports the argument that some prompt-level variation carries useful signal, though it is limited to a single dataset and a top-30% cutoff.
Experimental detail is extensive, with bootstrap intervals, hyperparameters, and conditioning protocols documented across multiple appendices. The authors state they "plan to release the scoring and analysis code, the scores and source identifiers of every scored response." Yet reproduction is currently blocked by the lack of released artifacts in the paper itself, undocumented training data for several models (marked "not stated" in Table 2), and deterministic single-run scoring that means "the intervals reflect sampling over prompts and not variation across runs."
Reward models score responses for preference optimization, filtering and ranking, and pooling scores across prompts needs them to be comparable. Yet pairwise evaluation compares two responses to the same prompt, so any score component shared by a prompt's responses cancels and goes untested. We study this component by comparing between-prompt variation in model scores with the same variation in human ratings of the same responses. We propose a human-referenced audit, built from a between-prompt variance share, a sibling-rating regression and prompt substitutions, and apply it to eighteen released scorers and six matched training conditions. We find that scores cluster by prompt more than the ratings do: on HelpSteer2, six reward models have between-prompt shares of 0.34 to 0.64 against a human share near zero. Replacing the original prompt with a fixed instruction keeps substantial clustering but weakens agreement with prompt-level human ratings, so clustering alone does not show agreement with humans. Part of that variation is useful: subtracting prompt means lowers the helpfulness of globally selected responses, and scores stay sensitive to a wrong prompt even when training shuffles prompt means. We propose that reward evaluation check between-prompt variation, sensitivity to a wrong prompt and usefulness for selection separately.
Pick a starting point or write your own. Challenges run in the background, so you can keep reading while the AI investigates.
No challenges yet. Disagree with the review? Ask the AI to revisit a specific claim.