HimmBO: High-Dimensional Interactive Model-Merging Bayesian Optimization for Image Generation
Fine-tuning diffusion models has yielded over 100K community adapters, but merging them interactively creates a high-dimensional, expensive black-box optimization problem: prior work requires live image generation in every round and is limited to a few dozen preselected adapters. HimmBO splits the search into a screening stage that recovers promising adapters via level-set estimation over a precomputed gallery, followed by a refinement stage that optimizes merge coefficients only within that low-dimensional active set. A style-aware Gaussian-process kernel transfers preference information between visually similar adapters, allowing the system to scale to libraries of over one thousand adapters while using under one tenth the image generations of prior work.
HimmBO presents a compelling, well-engineered extension of preferential Bayesian optimization to library-scale adapter merging. The two-stage decomposition and style kernel are principled, and simulated-user experiments up to $N = 1080$ show graceful degradation ($0.889$ final DreamSim similarity versus $0.907$ at $N = 40$) while competitors collapse. However, the evaluation rests almost entirely on a noise-free simulated user; as the authors acknowledge, "Our simulated user, however, provides noise-free and internally consistent feedback, whereas real users may be inconsistent or change their preferences during a session. A larger user study is needed to measure how often this occurs" (Sec. 6). Additionally, standard level-set estimation machinery is weakened because preferential utilities lack a common scale across rounds.
The strongest elements are the screening/refinement split and the style kernel. Decoupling active-set recovery from coefficient optimization is the right architectural response to high dimensionality, and the precomputed gallery eliminates image-generation latency during screening. The style kernel is not merely an embellishment: the authors show that "With a coefficient-only kernel, this feedback cost grows rapidly with library size. With the style kernel, it remains nearly flat and is about $130\times$ lower at $N = 1080$" (Sec. 4.2). Scalability ablations on nested libraries confirm that "HimmBO degrades gracefully with library size, reaching $0.889$ similarity at $N = 1080$, only $0.018$ below its $0.907$ at $N = 40$" (Sec. 5.1.2), while GimmBO falls to $0.762$ and Stylus+GimmBO to $0.828$.
The primary limitation is the reliance on simulated, noise-free preference feedback. DreamSim is used as a stand-in for both the style map and the user, creating an idealized loop that may overstate performance; the small preliminary real-user test mentioned in Appendix A.6 is insufficient to validate robustness. A second concern is the one-way transition between stages: as the authors note, "The transition from screening to refinement is also one-way, so an adapter omitted from the active set cannot be recovered later" (Sec. 6), which means screening errors are permanent. Finally, the LSE adaptation is incomplete because preferential feedback utilities are not on a common scale across rounds; the authors omit interval intersection and related stopping guarantees, leaving the screening stage without formal convergence properties.
The evidence supports the central scalability and efficiency claims within the simulated setting. Comparisons are reasonably fair: GimmBO is run with its published configuration, and the retrieval baseline Stylus+GimmBO is given an oracle text caption derived from the ground-truth target—an advantage unavailable to HimmBO. The ablation in Sec. 5.1.3 is thorough, isolating the effect of the style kernel, diversity term, and stick-breaking parameterization while fixing the screened active set. It finds that "Fixing the stick length to $B$, as in GimmBO, causes the largest loss by concentrating search near the cap rather than covering the broader range of total strengths seen in community merges" (Sec. 5.1.3), bolstering the case for the modified parameterization. A gap is that only linear merging is evaluated, and the style map assumes roughly additive visual effects that may fail under more elaborate merging rules.
Reproducibility is moderately strong but not complete. Implementation rests on standard libraries (BoTorch, GPyTorch, diffusers), and Appendix A documents hyperparameters in Table S1, including kernel priors, batch sizes, and acquisition weights. The authors state that "We will release the library manifest, curation pipeline, and test inputs" (Sec. 5.1), yet no code repository URL is provided. Independent reproduction would be blocked by the substantial engineering required to replicate the style map (DreamSim features, prompt-wise centering, PCA), the modified stick-breaking recursion, and the gallery precomputation pipeline. Additionally, the probit likelihood, Laplace approximation, and per-round hyperparameter refitting are described at a high level; lower-level numerical choices (e.g., L-BFGS-B multi-start count, jitter values) are relegated to the appendix and may still require clarification.
Fine-tuning has become a common way to customize text-to-image models, leading to community collections of over 100K lightweight adapters. Adapters sharing a base model can be merged with continuous coefficients. This turns a large collection into a high-dimensional continuous design space, which recent work explores interactively using preferential Bayesian optimization (PBO). However, existing approaches are limited to a few dozen preselected adapters and require expensive image generation in every interaction round. We introduce HimmBO, a two-stage method that scales preference-driven adapter merging to much larger collections while reducing image-generation cost. Its screening stage recovers the active adapter set through level-set estimation over pre-generated images, requiring no image generation. The refinement stage then optimizes merge coefficients in the recovered low-dimensional subspace. Both stages use a style-aware kernel, allowing preference information to transfer between visually similar adapters. In simulated-user image-matching experiments, HimmBO achieves higher final similarity than all comparisons while reaching a given similarity with under one tenth of the image generations required by prior work. Its performance remains stable as the collection grows to 1080 adapters, while competing methods degrade substantially.
Pick a starting point or write your own. Challenges run in the background, so you can keep reading while the AI investigates.
No challenges yet. Disagree with the review? Ask the AI to revisit a specific claim.