RA$^2$: Retain-Anchored Attraction Preserves Forget-Adjacent Utility in LLM Unlearning
LLM unlearning is usually tested by whether a model refuses harmful queries and retains clean-task accuracy. The authors highlight a gap: models can pass those checks while failing when benign prompts sit just outside the forgotten domain, a failure they term forget-adjacent utility loss. They propose RA2, which redirects forget-token hidden states toward nearby retain-supported anchors instead of simply pushing them away. Experiments on WMDP and MUSE show this directional choice meaningfully reduces neighborhood-level collateral damage.
RA2 identifies a genuine failure mode in latent-space unlearning and provides both a formal geometric diagnostic and a practical fix. The WMDP and direction-sweep experiments make a strong case that where forget representations land matters as much as whether they leave. That said, the method adds real engineering complexity—thousands of moving anchors with iterative nearest-neighbor lookups—and still incurs measurable drops in broad utility on some backbones, so the tradeoff is softened rather than eliminated.
The central insight, that purely repulsive forgetting can overshoot into unsupported representation regions, is well supported. Figure 1 and Table 3 show that RMU preserves ordinary MMLU yet collapses on MMLU-Perturbed and WMDP->MMLU, while RA2 stays near the original model on both. The directional sweep in Figure 3 is especially convincing: rotating the target into the repulsive half-plane sharpens direct forgetting but collapses local retain metrics, isolating direction as the causal factor.
The anchor field is costly: 5000 vectors with farthest-point sampling, momentum updates, and per-token top-k cosine-neighbor searches add wall-clock and memory overhead beyond simple gradient objectives. More importantly, the local geometric analysis in Section 4.1 assumes continuous tangent patches, while the actual optimization uses discrete anchors that may not satisfy the derived contract. On Qwen3-8B, RA2 still drops MMLU from 72.9 to 66.6, so broad retain utility is not fully preserved, and the harmful-cue drift gap $G_{\mathrm{cue}}$ remains positive at 3.580, indicating residual instability under harmful-adjacent cues.
The comparisons are internally consistent: all methods are evaluated under the same aligned WMDP profile and identical perturbation-based metrics. However, the forget-adjacent evaluations are purpose-built to probe exactly the inductive bias RA2 encodes, so they naturally favor the proposed method. The MUSE-Books results are more mixed—RA2 achieves the lowest verbatim memorization, but its retain knowledge score (55.36) lags well behind the retrained reference (69.10), suggesting that anchor-based redirection is not uniformly better across all unlearning contracts.
The authors release code and report key hyperparameters such as $m=5000$ anchors, $k=3$ nearest neighbors, and reshape layers $(\ell_r,\ell_t)=(7,8)$ for Zephyr and $(10,11)$ for Qwen3. Still, reproducing the anchor field requires implementing FPS initialization, momentum updates, and per-step stop-gradient assignment, which is more involved than standard gradient-ascent or NPO baselines. Because the full hyperparameter sheet is deferred to Appendix E, independent reproduction depends on reading that appendix to recover learning rates, temperatures, and warm-up schedules.
Large Language Model (LLM) unlearning is often evaluated primarily by target suppression and clean retain utility. These metrics can miss a local failure mode: a model may perform well on standard retain prompts, yet degrade on benign reasoning when prompts lie near the forgotten domain. This phenomenon is termed forget-adjacent utility loss. A fundamental question is: where should forget representations go after they leave the removed behavior? Successful unlearning depends not only on whether forget-conditioned representations depart from the harmful computation, but also on the direction in which they are redirected afterward. We propose RA$^2$ (Retain-Anchored Attraction), a latent-space unlearning method that redirects forget-token states toward nearby retain-supported representations while preserving higher-layer behavior on retain data. On WMDP and MUSE, RA$^2$ yields higher forget-adjacent utility than strong unlearning baselines at similar broad-utility levels. These results show that the migration direction of forget representations materially affects benign behavior near the forget boundary. Our code is available at https://github.com/ZacLing/retain_anchored_attraction.
Pick a starting point or write your own. Challenges run in the background, so you can keep reading while the AI investigates.
No challenges yet. Disagree with the review? Ask the AI to revisit a specific claim.