Skip to content

Commit b34acbd

Browse files
patrickvonplatenpcuenca
patrickvonplaten
and
pcuenca
authored
[SDXL Turbo] Add some docs (#5982)
* add diffusers example * add diffusers example * Comment about making it faster * Apply suggestions from code review Co-authored-by: Pedro Cuenca <[email protected]> --------- Co-authored-by: Pedro Cuenca <[email protected]>
1 parent 63f767e commit b34acbd

File tree

3 files changed

+173
-0
lines changed

3 files changed

+173
-0
lines changed

‎docs/source/en/_toctree.yml

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -72,6 +72,8 @@
7272
title: Overview
7373
- local: using-diffusers/sdxl
7474
title: Stable Diffusion XL
75+
- local: using-diffusers/sdxl_turbo
76+
title: SDXL Turbo
7577
- local: using-diffusers/kandinsky
7678
title: Kandinsky
7779
- local: using-diffusers/controlnet
@@ -333,6 +335,8 @@
333335
title: Stable Diffusion 2
334336
- local: api/pipelines/stable_diffusion/stable_diffusion_xl
335337
title: Stable Diffusion XL
338+
- local: api/pipelines/stable_diffusion/sdxl_turbo
339+
title: SDXL Turbo
336340
- local: api/pipelines/stable_diffusion/latent_upscale
337341
title: Latent upscaler
338342
- local: api/pipelines/stable_diffusion/upscale

‎docs/source/en/api/pipelines/stable_diffusion/sdxl_turbo.md

Lines changed: 53 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,53 @@
1+
<!--Copyright 2023 The HuggingFace Team. All rights reserved.
2+
3+
Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with
4+
the License. You may obtain a copy of the License at
5+
6+
http://www.apache.org/licenses/LICENSE-2.0
7+
8+
Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on
9+
an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the
10+
specific language governing permissions and limitations under the License.
11+
-->
12+
13+
# SDXL Turbo
14+
15+
Stable Diffusion XL (SDXL) Turbo was proposed in [Adversarial Diffusion Distillation](https://stability.ai/research/adversarial-diffusion-distillation) by Axel Sauer, Dominik Lorenz, Andreas Blattmann, and Robin Rombach.
16+
17+
The abstract from the paper is:
18+
19+
*We introduce Adversarial Diffusion Distillation (ADD), a novel training approach that efficiently samples large-scale foundational image diffusion models in just 1–4 steps while maintaining high image quality. We use score distillation to leverage large-scale off-the-shelf image diffusion models as a teacher signal in combination with an adversarial loss to ensure high image fidelity even in the low-step regime of one or two sampling steps. Our analyses show that our model clearly outperforms existing few-step methods (GANs,Latent Consistency Models) in a single step and reaches the performance of state-of-the-art diffusion models (SDXL) in only four steps. ADD is the first method to unlock single-step, real-time image synthesis with foundation models.*
20+
21+
## Tips
22+
23+
- SDXL Turbo uses the exact same architecture as [SDXL](./stable_diffusion_xl).
24+
- SDXL Turbo should disable guidance scale by setting `guidance_scale=0.0`
25+
- SDXL Turbo should use `timestep_spacing='trailing'` for the scheduler and use between 1 and 4 steps.
26+
- SDXL Turbo has been trained to generate images of size 512x512.
27+
- SDXL Turbo is open-access, but not open-source meaning that one might have to buy a model license in order to use it for commercial applications. Make sure to read the [official model card](https://huggingface.co/stabilityai/sdxl-turbo) to learn more.
28+
29+
<Tip>
30+
31+
To learn how to use SDXL Turbo for various tasks, how to optimize performance, and other usage examples, take a look at the [Stable Diffusion XL](../../../using-diffusers/sdxl_turbo) guide.
32+
33+
Check out the [Stability AI](https://huggingface.co/stabilityai) Hub organization for the official base and refiner model checkpoints!
34+
35+
</Tip>
36+
37+
## StableDiffusionXLPipeline
38+
39+
[[autodoc]] StableDiffusionXLPipeline
40+
- all
41+
- __call__
42+
43+
## StableDiffusionXLImg2ImgPipeline
44+
45+
[[autodoc]] StableDiffusionXLImg2ImgPipeline
46+
- all
47+
- __call__
48+
49+
## StableDiffusionXLInpaintPipeline
50+
51+
[[autodoc]] StableDiffusionXLInpaintPipeline
52+
- all
53+
- __call__

‎docs/source/en/using-diffusers/sdxl_turbo.md

Lines changed: 116 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,116 @@
1+
<!--Copyright 2023 The HuggingFace Team. All rights reserved.
2+
3+
Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with
4+
the License. You may obtain a copy of the License at
5+
6+
http://www.apache.org/licenses/LICENSE-2.0
7+
8+
Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on
9+
an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the
10+
specific language governing permissions and limitations under the License.
11+
-->
12+
13+
# Stable Diffusion XL Turbo
14+
15+
[[open-in-colab]]
16+
17+
SDXL Turbo is an adversarial time-distilled [Stable Diffusion XL](https://huggingface.co/papers/2307.01952) (SDXL) model capable
18+
of running inference in as little as 1 step.
19+
20+
This guide will show you how to use SDXL-Turbo for text-to-image and image-to-image.
21+
22+
Before you begin, make sure you have the following libraries installed:
23+
24+
```py
25+
# uncomment to install the necessary libraries in Colab
26+
#!pip install -q diffusers transformers accelerate omegaconf
27+
```
28+
29+
## Load model checkpoints
30+
31+
Model weights may be stored in separate subfolders on the Hub or locally, in which case, you should use the [`~StableDiffusionXLPipeline.from_pretrained`] method:
32+
33+
```py
34+
from diffusers import AutoPipelineForText2Image, AutoPipelineForImage2Image
35+
import torch
36+
37+
pipeline = AutoPipelineForText2Image.from_pretrained("stabilityai/sdxl-turbo", torch_dtype=torch.float16, variant="fp16")
38+
pipeline = pipeline.to("cuda")
39+
```
40+
41+
You can also use the [`~StableDiffusionXLPipeline.from_single_file`] method to load a model checkpoint stored in a single file format (`.ckpt` or `.safetensors`) from the Hub or locally:
42+
43+
```py
44+
from diffusers import StableDiffusionXLPipeline
45+
import torch
46+
47+
pipeline = StableDiffusionXLPipeline.from_single_file(
48+
"https://huggingface.co/stabilityai/sdxl-turbo/blob/main/sd_xl_turbo_1.0_fp16.safetensors", torch_dtype=torch.float16)
49+
pipeline = pipeline.to("cuda")
50+
```
51+
52+
## Text-to-image
53+
54+
For text-to-image, pass a text prompt. By default, SDXL Turbo generates a 512x512 image, and that resolution gives the best results. You can try setting the `height` and `width` parameters to 768x768 or 1024x1024, but you should expect quality degradations when doing so.
55+
56+
Make sure to set `guidance_scale` to 0.0 to disable, as the model was trained without it. A single inference step is enough to generate high quality images.
57+
Increasing the number of steps to 2, 3 or 4 should improve image quality.
58+
59+
```py
60+
from diffusers import AutoPipelineForText2Image
61+
import torch
62+
63+
pipeline_text2image = AutoPipelineForText2Image.from_pretrained("stabilityai/sdxl-turbo", torch_dtype=torch.float16, variant="fp16")
64+
pipeline_text2image = pipeline_text2image.to("cuda")
65+
66+
prompt = "A cinematic shot of a baby racoon wearing an intricate italian priest robe."
67+
68+
image = pipeline_text2image(prompt=prompt, guidance_scale=0.0, num_inference_steps=1).images[0]
69+
image
70+
```
71+
72+
<div class="flex justify-center">
73+
<img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/sdxl-turbo-text2img.png" alt="generated image of a racoon in a robe"/>
74+
</div>
75+
76+
## Image-to-image
77+
78+
For image-to-image generation, make sure that `num_inference_steps * strength` is larger or equal to 1.
79+
The image-to-image pipeline will run for `int(num_inference_steps * strength)` steps, e.g. `0.5 * 2.0 = 1` step in
80+
our example below.
81+
82+
```py
83+
from diffusers import AutoPipelineForImage2Image
84+
from diffusers.utils import load_image, make_image_grid
85+
86+
# use from_pipe to avoid consuming additional memory when loading a checkpoint
87+
pipeline = AutoPipelineForImage2Image.from_pipe(pipeline_text2image).to("cuda")
88+
89+
init_image = load_image("https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/cat.png")
90+
init_image = init_image.resize((512, 512))
91+
92+
prompt = "cat wizard, gandalf, lord of the rings, detailed, fantasy, cute, adorable, Pixar, Disney, 8k"
93+
94+
image = pipeline(prompt, image=init_image, strength=0.5, guidance_scale=0.0, num_inference_steps=2).images[0]
95+
make_image_grid([init_image, image], rows=1, cols=2)
96+
```
97+
98+
<div class="flex justify-center">
99+
<img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/sdxl-turbo-img2img.png" alt="Image-to-image generation sample using SDXL Turbo"/>
100+
</div>
101+
102+
## Speed-up SDXL Turbo even more
103+
104+
- Compile the UNet if you are using PyTorch version 2 or better. The first inference run will be very slow, but subsequent ones will be much faster.
105+
106+
```py
107+
pipe.unet = torch.compile(pipe.unet, mode="reduce-overhead", fullgraph=True)
108+
```
109+
110+
- When using the default VAE, keep it in `float32` to avoid costly `dtype` conversions before and after each generation. You only need to do this one before your first generation:
111+
112+
```py
113+
pipe.upcast_vae()
114+
```
115+
116+
As an alternative, you can also use a [16-bit VAE](https://huggingface.co/madebyollin/sdxl-vae-fp16-fix) created by community member [`@madebyollin`](https://huggingface.co/madebyollin) that does not need to be upcasted to `float32`.

0 commit comments

Comments
 (0)