PROBE: Diagnosing Residual Concept Capacity in Erased Text-to-Video Diffusion Models

The multi-level evaluation framework is comprehensive, combining classifier-based detection, CLIP semantic similarity, temporal reactivation curves, and human validation to robustly assess recovery. The empirical finding that erasure robustness correlates with intervention depth—weight-space unlearning (T2VUnlearning) showing stronger resistance than activation steering (SAFREE) or negative prompting (NegPrompt)—is consistent across all tested concept categories (objects, NSFW content, celebrities) and architectures (CogVideoX-2B/5B, Wan2.2-5B). The identification of temporal re-emergence represents a genuinely novel contribution specific to video generation, where frame-level metrics fail to detect progressive concept resurfacing across frames.

The interpretation of PROBE results as evidence of representational non-removal conflates recoverability via optimization with the existence of intact concept representations. While the frozen-parameter constraint ensures no new capacity is introduced during probing, successful optimization could exploit geometric properties of the latent space (e.g., proximity to non-erased concepts sharing visual attributes) rather than revealing dormant concept encodings. Human validation in Section VII-C shows discrepancies with automatic metrics for object and identity concepts, suggesting that some 'recovery' may not correspond to perceptually recognizable concepts. The theoretical analysis of temporal re-emergence hypothesizes propagation through temporal attention layers but provides no mechanistic evidence (e.g., attention head analysis) to support this causal claim.

The comparison across three intervention depths (input, activation, weight-space) provides strong evidence for the hierarchy of erasure robustness, aligning with theoretical expectations that deeper interventions should yield more persistent removal. The paper fairly positions its contribution relative to T2I predecessors like Ring-A-Bell and adversarial prompt search methods, noting that PROBE adapts textual inversion for diagnostic measurement rather than personalization and adds latent alignment to prevent co-occurring attribute collapse. The ablation study (Table XV) convincingly demonstrates that the latent alignment term ($\mathcal{L}_{\text{align}}$) consistently improves recovery rates over reconstruction-only probing, validating the design choice to anchor recovery to spatiotemporal structure. However, the comparison with P4D-K (Table X) showing PROBE achieves higher reactivation rates (28.90% vs 20.45%) on nudity concepts under T2VUnlearning is limited to a single configuration, and the claim that discrete prompt search is less suited to T2V models would benefit from broader architectural coverage.

The authors release code and provide detailed implementation details including hyperparameters (learning rate 0.02, $\lambda=1$, 5 pseudo-tokens, AdamW optimizer) and hardware requirements (NVIDIA H100). However, reproducibility is hindered by substantial computational requirements: training converges in 1k–3k steps requiring approximately 3–10 hours per concept on H100 GPUs depending on frame count. The sensitivity analysis shows results depend on alignment weight $\lambda$ (Table XV), reference data size (Table XVI shows degradation at 200 samples), and pseudo-token count (10 tokens worse than 5), indicating that exact reproduction requires careful tuning. The paper uses generated reference videos rather than external datasets, which standardizes the reference distribution but means reproduction requires first generating these from unerased models, adding computational overhead.