Questioning Instrumental Convergence in RLVR: Checkpoint Evidence from OLMo-3
Instrumental convergence predicts that capable agents develop power-seeking subgoals, but it is unclear whether reinforcement learning with verifiable rewards (RLVR) monotonically increases such behavior in language models. This paper evaluates OLMo-3-7B-Think across sixteen post-training checkpoints using InstrumentalEval and finds a nonmonotonic trajectory: instrumental-like outputs rise during early RLVR, peak near step 200, and then fall below the SFT baseline by the final checkpoint. The result challenges simple "more RL equals more convergence" narratives and shows that checkpoint-level monitoring can reveal dynamics hidden by endpoint comparisons.
The paper makes a carefully bounded but valuable empirical point: for OLMo-3-7B-Think, RLVR does not monotonically increase instrumental-convergence-like outputs on InstrumentalEval, and the final checkpoint falls significantly below the SFT baseline. The authors are appropriately cautious, framing the result as specific to this model family, benchmark, and judging protocol rather than a broad safety claim. However, the moderate inter-judge agreement and limited mechanistic insight mean the conclusion is stronger as a methodological demonstration than as a definitive safety assessment.
The checkpoint-trajectory design is the strongest feature; by evaluating thirteen intermediate RLVR steps rather than a single before/after comparison, the authors reveal dynamics that endpoint analyses would obscure. Their statistical treatment is also solid: paired item-level t-tests, Wilcoxon signed-rank tests, and a step-wise regression with a breakpoint at step 200 all support the claim that later RLVR reverses the early increase. The framing controls—per-checkpoint linear regression and INLP projection—add credibility by showing that a detectable linear framing direction does not explain the final SFT-to-RLVR gap.
The main limitation is external validity. Both evaluated models belong to the same OLMo-3 family, so the trajectory may reflect idiosyncrasies of this pipeline—its verifier design, reference-policy constraints, or base-model distribution—rather than RLVR in general. The preliminary 32B results only compare endpoints and do not establish whether the same nonmonotonic arc holds at scale. A second concern is judge reliability. Average inter-judge agreement between GPT-4o and Gemini 2.0 Flash is only 68.75%, dropping to 55.26% near the peak IC checkpoint, which raises questions about whether the early-rise phase reflects stable behavior or noisy classification. The paper also cannot distinguish "reduced expression of deceptive reasoning" from "reduced probability of verbalizing that reasoning," leaving the inner-alignment implications unresolved.
The evidence supports the bounded claim that final RLVR reduces InstrumentalEval scores relative to SFT for this model, and the authors fairly situate their checkpoint-trajectory analysis as an extension of He et al.'s original endpoint comparison rather than a refutation of it. They acknowledge that narrow verifiers may limit opportunities for broad instrumental strategies without solving outer alignment, and they do not overstate dissimilarity with prior work on deceptive alignment. The comparison would be strengthened by a more explicit quantitative or qualitative discussion of how the OLMo-3 trajectory maps onto He et al.'s findings across different training paradigms.
Reproducibility is mixed. The evaluation relies on publicly released OLMo-3 checkpoints and the open InstrumentalEval benchmark, and the judge prompt is provided in Appendix Table 9. However, the authors do not release evaluation code, exact API versions, or full hyperparameters for the INLP and regression analyses. Independent reproduction is therefore feasible in principle but would require reimplementing the judging pipeline and could be blocked by the same moderate inter-judge variance.
Instrumental convergence is often invoked as a general reason to expect capable agents to acquire or express power-seeking subgoals: self-preservation, resource acquisition, deception, or resistance to oversight which poses an important threat to the society. For LLMs, reinforcement learning is the first stage of post-training that directly optimizes behavior against an objective; therefore, it is the stage where convergent instrumental behavior, on IC grounds, likely to appear. Does reinforcement learning with verifiable rewards (RLVR) make a model more likely to express instrumental-convergence behavior, or does this depend on the training regime, evaluation frame, and stage of post-training? We study this question as a checkpoint-conditioned behavioral trajectory rather than as a single before/after safety score. Using InstrumentalEval, we evaluate OLMo-3-7B-Think across SFT, DPO, 13 intermediate RLVR checkpoints, and the final RLVR model. The trajectory is nonmonotonic: instrumental rate rises from 25.0% at SFT to 46.1% near RLVR step 200, then falls to 11.8% at the final checkpoint. Paired item-level tests and step-wise regression support the final SFT-to-RLVR reduction. Prompt framing does not explain this pattern: a harmful-prompt indicator combined with other framing effects explains only 15% of within-checkpoint score variance on average, and Iterative Null-space Projection (INLP) removes a detectable framing direction without changing the SFT-to-final-RLVR gap. Together with preliminary 32B endpoint results (35.53% SFT to 22.37% RLVR final), these findings indicate that RLVR does not monotonically increase instrumental behavior in this setting: the rate rises early, then falls below the SFT baseline. The result is specific to this model family and benchmark, and judge agreement is moderate, so absolute rates should be read cautiously.
Pick a starting point or write your own. Challenges run in the background, so you can keep reading while the AI investigates.
No challenges yet. Disagree with the review? Ask the AI to revisit a specific claim.