DANGERLENS: Probing Dangerous Representations Hidden by Safety Alignment
DangerLens asks whether two language models with identically safe refusal behavior can still differ in how much hazardous knowledge remains internally accessible. The authors train paired Qwen1.5 checkpoints—one directly safety-aligned, the other first exposed to harmful data then aligned to the same standard—and introduce a vocabulary-space probe that uses a Jacobian lens and blinded semantic scoring to count harmful goals and operational means inside hidden states. The work matters because current safety audits rely on surface behavior alone, leaving latent dangerous representations undetected before downstream fine-tuning or attacks.
The paper presents a genuinely clever controlled experiment with useful ablations and multiscale replication, but its central quantitative evidence rests on an undisclosed LLM semantic judge whose biases and safety training are unknown. While the authors diligently human-audit a 700-pair sample and run robustness checks on transport and inventory choices, the fact that every primary count is produced by an automated judge mentioned only in the ethics section is a severe methodological transparency gap. The claim that prior exposure leaves greater hazardous concept accessibility is conceptually important but operationally reduced to the outputs of an uncharacterized black-box classifier.
The paired-path design is the clearest strength. By matching the final safety alignment phase and later harmful re-exposure dose, the authors isolate training history as the sole difference and show that previously exposed checkpoints break faster, corroborating the internal probe with behavioral trajectories. The ablation suite is equally strong: Jacobian transport consistently outperforms identity transport, shared reference maps still reveal the gap, and reducing the candidate budget to Top-10 preserves positive contrasts, which together suggest the measured difference is not an artifact of the readout machinery. A blinded human audit confirming 97.3% of explicit hazard terms on 700 sampled pairs further calibrates the pipeline.
The most serious flaw is hidden in a single ethics-section sentence: the entire semantic pipeline—relevance scoring, role assignment, and ultimately the counts $C_{iM\ell p}$ and $\bar{C}_{iM\ell}$—is executed by an unspecified LLM. Because the main human audit validates only a small sample and the judge model is never named, readers cannot assess whether the measured contrast reflects genuine representational differences or the judge's own safety-refusal patterns and idiosyncratic biases.
A second concern is the depth-band selection procedure: the primary window $[0.45,0.90]$ was fixed after inspecting a pilot run, creating a risk that the window was tuned to maximize the expected separation before the confirmatory runs were analyzed. Finally, the behavioral trajectories at higher re-exposure doses ($q \ge 800$) often converge or become noisy, weakening the implied causal link between internal accessibility counts and future susceptibility.
The paper supports its core claim that prior-exposure checkpoints show higher internal dangerous-concept counts, and the dimension-resolved profiles (positive on D2, D4, D6 across scales) lend the finding semantic specificity. However, the authors do not benchmark against simpler mechanistic alternatives—such as linear refusal probes or the single refusal direction reported by Arditi et al.—so it is unclear whether DangerLens adds detection power beyond known representational signatures. Comparisons to Qi et al. and Ji et al. on fine-tuning fragility are accurate but largely narrative; the paper would be stronger with a direct head-to-head showing that Jacobian-lens counts predict susceptibility better than a baseline probe. Notably, the supplementary held-out association between pre-update internal contrast and post-update behavioral change is relegated to an appendix and not integrated into the main evidence chain.
Reproducibility is mixed. Hyperparameters, training settings, and software versions are documented in unusual detail in Appendix E, and the use of open base weights (Qwen1.5) helps. Yet several critical components are missing: no public code repository URL is provided, the identity of the LLM semantic judge is undisclosed, and the custom DLAB-SPP dataset and 800-prompt evaluation bank are not released—without them, independent auditors cannot reconstruct the probe. The authors state that fine-tuned checkpoints and harmful completions will be kept as controlled artifacts, which is responsible but further limits reproduction. More broadly, exact judge prompts and the full surface-normalization rules are deferred to appendices that are truncated in this preprint, leaving the measurement protocol incompletely specified.
Safety alignment teaches language models to refuse harmful requests, yet the internal accessibility of hazardous concepts can retain traces of earlier training. Can checkpoints with equally safe behavior differ in what remains accessible behind their refusals? In this work, we investigate this question through paired training histories and a direct audit of intermediate representations. We introduce DANGERLENS, a probe that maps hidden states into vocabulary space and uses blinded semantic judgments to distinguish harmful goals and operational means from hazard recognition and safety control. Experiments on three Qwen1.5 scales show that checkpoints with prior harmful exposure generally retain greater hazardous concept accessibility after the same safety alignment, with the strongest separation in intermediate and later layers. The same harmful update also produces different behavioral trajectories from closely matched starting points. Human relevance validation and readout ablations reinforce the measured accessibility contrast. Our findings show that training history remains visible inside behaviorally matched models and motivate internal accessibility audits alongside output safety evaluations.
Pick a starting point or write your own. Challenges run in the background, so you can keep reading while the AI investigates.
No challenges yet. Disagree with the review? Ask the AI to revisit a specific claim.