Joint Branch-Space Transform Coding for Diffusion Activation Quantization with Classifier-Free Guidance
Diffusion models with classifier-free guidance (CFG) compute paired conditional and unconditional activations at each denoising step, yet existing post-training quantizers treat those branches as independent sources. This paper identifies the CFG branch space as a structured coding axis—matched activations are strongly correlated and jointly shaped by the guidance direction—and proposes Guidance-Correlation Branch Transform (GCBT), a lightweight $2\times 2$ orthogonal rotation applied per layer before activation quantization. It admits a closed-form per-layer solution, plugs into existing PTQ pipelines with negligible overhead, and offers a complementary, easy-to-adopt lever for low-bit generation fidelity.
GCBT is a well-motivated and practically useful addition to the diffusion PTQ toolbox. The observation that native conditional/unconditional coordinates leave quantization fidelity on the table is supported by both layerwise statistics and controlled ablations. The proposed rotation derives cleanly from a guidance-weighted surrogate objective, and the experiments show consistent—if sometimes modest—gains across SVDQuant, Q-DiT, OrbitQuant, and ViDiT-Q hosts on image and video models. The authors are careful with paired statistical testing and acknowledge that the closed-form optimum is for a surrogate, not the end-to-end quantizer.
The strongest aspect of the paper is the empirical breadth and statistical rigor. The authors evaluate GCBT on multiple image and video diffusion architectures (PixArt-$\Sigma$, SANA-1.6B, SD3.5-Large, Wan2.1, CogVideoX, OpenSora) across four distinct PTQ backends, showing that branch-space gains transfer broadly. They report paired 95% confidence intervals and explicitly note when effects are inconclusive rather than inflating claims. The system overhead is convincingly small—only 1–2% latency increase and no memory change—because the transform is a per-layer $2\times 2$ rotation applied only across the CFG branch axis.
The theoretical guarantee is limited to a surrogate. Theorem 3.3 optimizes the equal-rate quantization-noise surrogate in Eq. 7, which assumes diagonal, equal-rate distortion and ignores cross-coordinate quantization-error terms; the authors concede in Appendix C that this means the rotation is optimal for the surrogate, not necessarily for the actual host quantizer or end-to-end generation error. In the experiments, gains are large under aggressive W4A4 SVDQuant but shrink markedly under Q-DiT W4A8, where several comparisons are statistically inconclusive and absolute deltas are tiny. The video results also show mixed significance on VBench imaging quality (I.Q.) and teacher PSNR for some models, suggesting the benefit is not uniform. Finally, the comparison to CFG-aware calibration relies on a self-implemented GAMP-inspired method because the original code is unavailable, weakening claims of complementarity.
The evidence generally supports the core claim that branch-space rotation improves fidelity, but the comparisons could be deeper. The paper positions GCBT as complementary to existing timestep-, feature-, and layer-aware methods, yet does not compare against recent rotation-based diffusion quantizers such as TR-DQ and OrbitQuant in a unified table—only against the native host. While Proposition 3.4 argues algebraic compatibility with shared feature-space rotations, an ablation that actually composes GCBT with TR-DQ or OrbitQuant’s native rotations is absent. The branch-basis ablation (Table 4) does show that combining guidance and geometry outperforms either alone, which is helpful, but the full factorial of host x rotation x guidance scale is not explored beyond the single PixArt-$\Sigma$/SVDQuant setting.
Reproducibility is mixed. The paper provides detailed generation and calibration hyperparameters in Appendix B.4, uses fixed paired prompts and seeds, and reports bootstrap confidence intervals, which aids statistical reproducibility. However, no code, data, or pre-computed rotation matrices are released or promised, and several key host implementations require external pipelines (SVDQuant via Nunchaku, OrbitQuant). Without open-source release of the GCBT layer wrapper and calibration scripts, independent reproduction would require re-implementing the branch-rotation logic and integrating it into each host quantizer. The 64-sample calibration is small, but the exact prompt sets and measurement protocol for $\Sigma_\ell$ are not publicly hosted.
Post-training quantization for diffusion models increasingly exploits timestep, feature, and layer structure. While recent work has begun incorporating CFG structure into diffusion quantization, activation quantization still operates independently across conditional and unconditional coordinates, leaving cross-activation structure unexploited. We show that matched CFG activations form a strongly correlated two-dimensional source and that, under a fixed bit budget, the choice of branch coding basis materially affects quantization fidelity. Motivated by this observation, we introduce branch-space transform coding, which rotates matched CFG branches via an offline derived 2x2 orthogonal matrix, requiring minimal modifications to model parameters or the quantization pipeline. We further derive the Guidance-Correlation Branch Transform (GCBT), which jointly incorporates the CFG guidance direction and cross-branch second moments. Under an equal-rate quantization-noise surrogate, GCBT admits a closed-form per-layer solution without gradient optimization or angle search. Applied on top of existing diffusion PTQ methods, GCBT yields statistically significant fidelity gains in most evaluated comparisons with no statistically significant degradation, while leaving the underlying host quantization pipeline unchanged.
Pick a starting point or write your own. Challenges run in the background, so you can keep reading while the AI investigates.
No challenges yet. Disagree with the review? Ask the AI to revisit a specific claim.