iADD: Improving Alignment and Diversity in Diffusion Policy Optimization
iADD tackles the alignment-diversity trade-off in reinforcement-learning finetuning of diffusion models such as Stable Diffusion. Its core idea is to update early denoising timesteps rather than late ones, combined with a sparse incremental curriculum and Feynman–Kac particle branching during training. Better alignment without mode collapse matters because existing reward-optimization methods frequently degrade visual quality and diversity.
The paper makes a credible case that early-timestep updates and FK-based path guidance improve the reward–diversity frontier in diffusion RL. The main empirical gains are solid, especially on rare prompts, and the ablation in Figure 6 directly links the theoretical predictions to measurable Jacobian log-volumes. However, the theory leans on idealized deterministic-flow and first-order assumptions, so the strong claim that late-timestep updates are “harmful for diversity” reads slightly stronger than the evidence supports.
The clearest strengths are the explicit connection between timestep selection and generative volume, plus the accompanying empirical validation. Proposition 1 predicts larger output perturbations from early or uniform updates, and Figure 6 shows that during real training the cumulative $\|\delta x_0\|$ and the full-Jacobian log-determinant both favor sparse early updates, matching the theoretical prediction. In downstream metrics this translates to higher Inception Score and a large rarity gap over baselines.
First, the theoretical guarantees are comparative and idealized. The proofs assume a deterministic DDIM flow and rely on first-order sensitivity and Lipschitz bounds; the main text concedes that “the comparative conclusion only requires the corresponding Lipschitz sensitivity ... to be larger for earlier, high-leverage timesteps,” so the results are predictions rather than training-time bounds. Second, checkpoint selection introduces optimism bias: methods are reported at “the checkpoint closest to the ideal (1,1) on the Reward-Inception Score Pareto frontier,” which means reported numbers are post-hoc selected rather than from a fixed stopping rule. Third, the FK branching mechanism adds compute cost that is acknowledged in Table 1 but never quantified in wall-clock time or total FLOPs, making the practical trade-off unclear.
The evidence in Table 3 supports the claim that iADD dominates DDPO and B2-DiffuRL across image generation, vanishing-point correction, and 3D scene synthesis on both reward and diversity metrics, with a particularly large rarity margin. Table 4 offers a nuanced picture: recent GRPO variants (DanceGRPO, BranchGRPO) can edge iADD in raw CLIPScore, but iADD achieves higher IS, rarity, and AUC, and combining iADD with GRPO yields the best overall numbers. This shows the proposed components are complementary rather than obsoleted by newer policy-optimization flavors. The comparison is generally fair because the same pretrained backbone, LoRA rank, and prompt sets are used, though the operating-point selection caveat applies uniformly.
Reproducibility is mixed. On the plus side, the authors provide a project page and GitHub repository, and the supplement gives detailed hyperparameters including LoRA configurations, the incremental schedule {5,10,15,20} timesteps over four stages, FK branching at t_b=5 with 4 particles, resampling timesteps, and the exact CLIP- and VP-reward algorithms. On the minus side, the paper does not report total training time or wall-clock cost for the FK branching, random seeds are not specified, and the 3D experiments rely on a custom continuous-domain variant of MiDiffusion whose exact preprocessing and floor-plan encoder weights would need to be released to enable independent reproduction. The deterministic evaluation seeds and full prompt lists in supplementary Table S6 help, but without a fixed stopping rule the reported operating-point metrics may be hard to replicate exactly.
Reinforcement learning based post training of diffusion models, such as Denoising Diffusion Policy Optimization (DDPO), optimizes a reverse diffusion process under a reward function. However, current approaches to reward optimizations do so at the cost of diversity and quality. In this paper, we provide better tradeoffs through careful theoretical considerations and method design. We analyze the theoretical framework and mathematically demonstrate that \emph{only-latter timestep} updates of diffusion model may be harmful for diversity contrary to the conclusions presented in a previous work. Additionally, we propose an incremental Feynman-Kac training based on strong theoretical foundations in order to achieve the best-yet alignment-diversity tradeoffs. We perform extensive experiments and compare our method against related diffusion policy optimization approaches in three different tasks and also provide strong ablations for each component, thus validating strong performance gains in both alignment and diversity.
Pick a starting point or write your own. Challenges run in the background, so you can keep reading while the AI investigates.
No challenges yet. Disagree with the review? Ask the AI to revisit a specific claim.