Debias Anything: Fairness with Diversity without Supervision in Diffusion Models
Diffusion models amplify demographic biases present in training data, and current debiasing methods usually require costly attribute annotations, model retraining, or sacrifice output diversity. This paper proposes Debias Anything, a training-free guidance method that steers both unconditional and text-conditional diffusion models toward arbitrary target proportions for any sensitive attribute described in free text. By linking the frozen denoiser to a vision-language embedding space via a lightweight adapter, it jointly enforces batch-level fairness through optimal assignment and restores diversity through a semantic disagreement score, offering broad applicability without labeled data.
The paper presents a compelling unsupervised alternative to supervised debiasing. Its adapter-based guidance closes the performance gap with attribute-supervised baselines while operating only on text prompts, and the per-step Hungarian assignment is theoretically elegant. However, the reliance on vision-language encoder linearity and the narrow experimental scope temper the claim of universal applicability. The authors themselves note that "Guidance can only push towards images the model can produce: on Stable Diffusion, male nurses are rare and remain hard to reach." (Section 6) Yet they also show that "Debias Anything generates the image proportion closer to the target than a classifier trained on labeled images, and with the diversity term it gives a lower or comparable FID at comparable discrepancy." (Section 5.1)
The simulation-free adapter that predicts clean semantic embeddings from noisy inner activations is well motivated and elegantly avoids out-of-distribution issues. The method convincingly unifies fairness and diversity: the batch-level fairness term with per-step optimal assignment is theoretically clean, and the diversity term transposes SGMS into the encoder space at no extra cost. The authors argue for broad compatibility: "The presented method is independent of how the model is conditioned, applies to unconditional and text-conditional models alike, and composes with any other guidance already in use." (Section 4.3) They also clearly position their contribution: "Our rarity term transposes SGMS into the encoder's space through the same adapter for efficiency purposes, and it is added to the fairness term so that the tail is recovered at the target proportion." (Section 2)
The theoretical justification for the classifier approximation relies on Lemma 1, which bounds the gap between the ideal classifier and the tractable surrogate by $e^{2w} w^2 V_k(x)/2$, but the paper never measures this variance empirically during sampling. This leaves the bound's tightness unclear at high noise levels, precisely when guidance must act. As noted, "The gap between the ideal classifier and ours therefore vanishes when $V_k$ approaches $0$, that is, once the model has decided the attribute." (Section 4.2)
Experimental validation is narrower than the marketing suggests: all results focus on human faces with uniform target proportions and single seeds, and steering one attribute inadvertently moves others. The authors candidly admit these limits: "Our evidence is on human faces only, at uniform target proportions, with single seeds on CelebA-HQ and Stable Diffusion." (Section 6)
The empirical evidence largely supports the core claim that unsupervised text-prompt guidance can match supervised baselines. On CelebA-HQ, Debias Anything achieves lower discrepancy than Balancing Act with better or comparable FID; on Stable Diffusion, it outperforms several attribute-specific methods despite using no labels. Comparisons are generally fair and transparently qualified. The authors note a reproduction caveat for Score Guidance: "For Score Guidance, the numbers we obtain differ from those reported in the original paper. We reproduced its attribute classifier, and we believe the gap comes from the reference set behind FID." (Section 5.1) They also correctly contextualize when DiffLens wins: "Where DiffLens is ahead, on gender and age, it is advantaged by attribute knowledge: labeled images locate the features that carry the attribute inside the network, and it edits those features directly." (Section 5.1)
Reproducibility is reasonably well addressed: the authors release code, provide appendices on adapter architecture and training, and report metric definitions. However, independent reproduction faces practical hurdles. The adapter must be retrained per generator backbone, and guidance is sensitive to hyperparameters: "The two guidance weights have a narrow useful range that we select by sweeps as detailed in the appendix." (Section 6) Evaluation also depends on specific external classifiers whose exact preprocessing is not fully specified in the main text. The repository is provided: "Our code is available at https://github.com/theaudaudiffret/Debias_anything." (Reproducibility statement)
Although diffusion models produce high-quality images, they also reproduce and amplify demographic imbalances in their training data. Debiasing their generation process post-training w.r.t. some sensitive attribute usually relies on classifier guidance or explicit text extra-conditioning, but this reduces methods' applicability and output diversity. Conversely, methods promoting diversity alone do not ensure fair attribute representation. In this paper, we propose a method tackling fairness and diversity jointly that is generally applicable to any diffusion model and any sensitive attribute. To this end, an adapter connects the frozen diffusion model to a pretrained vision-language embedding space, enabling fairness and diversity guidance without sensitive-attribute annotations. For fairness, pairs of text prompts define attribute directions which guide batch composition towards specific proportions. For diversity, we introduce a score measuring disagreement between the semantic estimates derived from this representation. The formulation supports unconditional and text-conditional diffusion models, while requiring no prior knowledge or data of sensitive attribute. Experiments confirm that our method improves quality and diversity scores at comparable fairness levels.
Pick a starting point or write your own. Challenges run in the background, so you can keep reading while the AI investigates.
No challenges yet. Disagree with the review? Ask the AI to revisit a specific claim.