New research: SeedSelect & norm-guided latent space exploration

[catboxanon]

Two recent papers that claim to improve overall image generation and assistance in generating rarer concepts.

It is all about where you start: Text-to-image generation with seed selection
Paper: https://arxiv.org/abs/2304.14530

[...] We show that rare concepts can be correctly generated by carefully selecting suitable generation seeds in the noise space, a technique that we call SeedSelect. SeedSelect is efficient and does not require retraining the diffusion model. We evaluate the benefit of SeedSelect on a series of problems. First, in few-shot semantic data augmentation, where we generate semantically correct images for few-shot and long-tail benchmarks. We show classification improvement on all classes, both from the head and tail of the training data of diffusion models. We further evaluate SeedSelect on correcting images of hands, a well-known pitfall of current diffusion models, and show that it improves hand generation substantially.

Norm-guided latent space exploration for text-to-image generation
Paper: https://arxiv.org/abs/2306.08687

[... diffusion model's] latent seed space is still not well understood and has been shown to have an impact in generating new and rare concepts. Specifically, simple operations like interpolation and centroid finding work poorly with the standard Euclidean and spherical metrics in the latent space. This paper makes the observation that current training procedures make diffusion models biased toward inputs with a narrow range of norm values. This has strong implications for methods that rely on seed manipulation for image generation that can be further applied to few-shot and long-tail learning tasks. To address this issue, we propose a novel method for interpolating between two seeds and demonstrate that it defines a new non-Euclidean metric that takes into account a norm-based prior on seeds. We describe a simple yet efficient algorithm for approximating this metric and use it to further define centroids in the latent seed space. We show that our new interpolation and centroid evaluation techniques significantly enhance the generation of rare concept images. This further leads to state-of-the-art performance on few-shot and long-tail benchmarks, improving prior approach in terms of generation speed, image quality, and semantic content.

The 1st paper covers a new method to introduce better initial seeds, what is essentially a few-shot learner. The 2nd paper proposes a new interpolation method they refer to as NAO-centroid (NAO standing for Norm-Aware Optimization), which improves upon more commonly used interpolation methods (for reference, the web UI uses slerp), and as a bonus improves the SeedSelect procedure's speed by 10x.

Code and example images for both papers are available here (this is NOT an extension): https://github.com/dvirsamuel/SeedSelect