Scaling Laws for Safety Probes
The paper asks how to predict the performance of a safety probe before it is trained. Its core finding is that probe error follows two scaling laws: a power-law floor in model size and a Hill-curve approach to that floor as labels increase. This matters because it could let safety teams budget labels and choose architectures for new models without expensive grid search.
The paper delivers a broadly convincing empirical framework for predicting probe error from model size and label count. The sweep across 20 base models, five families, and six tasks is unusually thorough for monitoring research. However, the abstract oversells architectural universality by claiming that only fitted constants differ across probe architectures, when the last-token probe in fact requires a different functional form with an extra steepness parameter $g$. More importantly, the laws are fitted on base models while deployed monitors read post-trained models, and the authors concede this gap is unresolved.
The two-law model captures the measurements remarkably well. The minimal-pair dataset design avoids the saturation that plagues public safety benchmarks, and the fits are tight across most families and tasks. Extrapolation from smaller models works: the laws predict the next model’s probe error to within $0.02$ in $72\%$ of cases and within $0.05$ in $97\%$.
First, the abstract claims the laws hold across architectures with only constants changing, yet Section 4.4 shows the last-token probe needs Eq. 3 with an added steepness $g=0.61$, altering the functional form itself. This is not merely a different constant.
Second, the study is limited to base models. Appendix A explicitly states that "The laws are fitted on base models, while a deployed monitor reads a post-trained one," and the limited post-training data available shifts the fitted exponent. The practical tool the paper promises therefore lacks validation on the actual deployment target.
Third, the cost model used to choose between more labels and a larger model---parameters times labels---is a crude proxy that ignores actual labeling prices, inference latency, and serving constraints.
The evidence is extensive: 20 models, five families, two architectures, and $10^6$ labels per dataset. Comparisons to Hoffmann et al. and Kaplan et al. on the same cells are fair. Yet the datasets are synthetic minimal-pair constructions, and the authors note that the law fits less well on Policy violation because its labels are noisier. This suggests generalization to naturalistic, real-world safety data remains an open question.
The authors describe data construction and probe architectures in detail, but there is no mention of releasing code, the constructed datasets, or the trained probes. Key hyperparameters (regularization strength, layer selection per seed, smoothing for CC++) are relegated to appendices not fully shown in the available text. Moreover, reproducing the full results would require running thousands of probe fits across 20 large language models, a substantial compute barrier.
Safety probes trained on a model's activations are a standard tool for monitoring language models in real time. Each time a new model is released or a new harm area needs to be monitored, the probe must be retrained, which requires new labels and a choice of probe architecture. Yet neither the model nor the labels tell us in advance how well the resulting probe will perform. Finding out requires training and comparing many probe configurations. In this paper, we ask whether probe performance can instead be predicted in advance from model size and the number of labels. Existing safety datasets are insufficient for this, as they are either limited in scale or already saturated. We therefore construct six new safety datasets of sufficient size and difficulty. On these datasets, we train two probe architectures on 20 base models from five model families, with 32 to $10^{6}$ labels for each model. We measure performance using probe error, defined as 1 - AUROC, where lower values are better. We find that probe error follows two simple scaling laws. First, the lowest probe error a model can reach falls as a power law in its size. Second, probe error decreases towards its floor as the number of examples increases; the slope increases as the target model size increases. Both laws hold across tasks, model families and probe architectures, and only their fitted constants differ. When fitted on the smallest models of a family, the laws predict the error of the next larger model to within 0.02 in 72% of cases. They also identify whether more labels or a larger model is the cheaper way to reach a target error in 73% of cases. Finally, they choose the probe architecture with an extra error of only 0.006 on average when they pick the worse one. These laws allow safety researchers to determine how to create the best probe for any given model and label budget.
Pick a starting point or write your own. Challenges run in the background, so you can keep reading while the AI investigates.
No challenges yet. Disagree with the review? Ask the AI to revisit a specific claim.