A Safe Prototype Is Not a Safety Direction: Reference Dependence and Prompt Confounds in Response-Safety Embeddings
Can response safety be scored by cosine similarity to a safe-response embedding centroid? The paper shows that such a raw positive prototype identifies a location, not a separating direction, and formalizes why positive-only observations cannot determine the sign of the safe-unsafe gap. On three prompt-controlled safety corpora and four frozen encoders, the unreferenced prototype is weak or inverted while an explicit safe-minus-unsafe reference recovers useful ranking; the result is carefully scoped to the centroid rule, not all one-class methods.
The paper presents a narrow, rigorous critique that holds up well within its stated bounds. The geometric argument in Proposition 1 is elementary but correctly frames why positive-only observations cannot identify the sign of the safe-unsafe gap. The empirical audit is carefully controlled, using prompt-grouped splits, exact deduplication, multiple encoders, nuisance baselines, and family-wise testing, and the conclusion is appropriately scoped.
The formal argument and the controlled empirical design are the strongest elements. Proposition 1 shows that for any safe distribution on the unit sphere with $0<\|\mu_s\|_2<1$, the sign of the expected score gap is not identified. The audit's prompt-control structure is exemplary: by hashing canonical prompts to splits and using mixed-label prompts or native discordant pairs, the study avoids the prompt-composition confounds that plague many safety evaluations. The inclusion of scalar nuisances such as log response length and stylistic features further anchors the embedding readouts to realistic baselines.
The narrow scope limits actionable takeaway: the safe-prototype rule is shown to fail on the specific frozen encoders and English datasets tested, but the paper does not deeply explore whether the failure persists with safety-tuned encoders, different pooling strategies, or non-cosine similarity metrics. BeaverTails proves difficult for everyone: even the referenced direction accepts very few safe responses at a 5% false-safe target on that corpus, suggesting these readouts are weak diagnostics there regardless of reference. The dramatic inversion on the Aegis jury control is a useful robustness check but explicitly not a human replication.
The evidence robustly supports the central claim that an unreferenced positive centroid is unreliable, and comparisons to related work are deliberately modest. The author contextualizes the result against one-class and positive-unlabeled learning, and evaluates difference-of-means, two-centroid, mixture-centered, RankOne, and supervised linear readouts on the same frozen embeddings, showing that signal exists in the representations but the prototype misorients it. The paper is careful not to overstate its conclusions.
Reproducibility is strong in some dimensions and limited in others. The artifact includes code, aggregate metrics, row-order hashes, source manifests, and pinned dataset revisions. The author reports protocol tests that enforce exact reproduction of primary AUC values, and hyperparameters are specified for the supervised baseline, but thresholds are corpus- and method-specific and must be recalibrated for any independent deployment. A notable limitation is that exploratory compute and embedding time were not fully logged.
Can response safety be scored by cosine similarity to the mean embedding of known-safe responses? A recent sleeper-agent detector proposes exactly this score, yet the raw positive-centroid rule is not identified: positive observations locate the safe class relative to an encoder origin, but do not determine which direction separates safe from unsafe responses. We audit the rule on two prompt-controlled, human-labeled corpora and one auxiliary jury-labeled source control, using four frozen encoders and prompt-grouped splits. On the human-labeled corpora the safe prototype reaches ROC-AUC 0.457-0.545, with two cells significantly below chance and one above, while an explicit safe-minus-unsafe reference reaches 0.588-0.738 on the same embeddings; on the jury control the prototype is inverted (0.358-0.405) and the reference reaches 0.754-0.793. At validation-calibrated 5% false-safe thresholds, the reference accepts more safe responses on PKU-SafeRLHF (0.153-0.263 versus 0.039-0.061 across encoders) and Aegis (0.189-0.291 versus 0.004-0.045), but not reliably on BeaverTails. A fully unlabeled held-out reference recovers part to most of the referenced ranking, much less when only 5% of the pool is unsafe, whereas 80-634 labeled unsafe responses recover most of it. Prompt-only ablations show that prompt-label composition can inflate uncontrolled evaluations. This is a bounded result about a raw positive centroid, not all one-class methods or safety-specialized guards. A class mean is a location, not necessarily a safety direction; a declared reference with enough unsafe mass identifies orientation.
Pick a starting point or write your own. Challenges run in the background, so you can keep reading while the AI investigates.
No challenges yet. Disagree with the review? Ask the AI to revisit a specific claim.