Skip to content

Commit d399d5e

Browse files
stevhliustevhliu
committed
draft
1 parent f064b3b commit d399d5e

File tree

3 files changed

+124
-50
lines changed

3 files changed

+124
-50
lines changed

‎docs/source/en/_toctree.yml‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -64,6 +64,8 @@
6464
title: Overview
6565
- local: using-diffusers/create_a_server
6666
title: Create a server
67+
- local: using-diffusers/batch_inference
68+
title: Batch inference
6769
- local: training/distributed_inference
6870
title: Distributed inference
6971
- local: using-diffusers/scheduler_features

‎docs/source/en/using-diffusers/batched_inference.md‎

Lines changed: 122 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,122 @@
1+
<!--Copyright 2025 The HuggingFace Team. All rights reserved.
2+
3+
Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with
4+
the License. You may obtain a copy of the License at
5+
6+
http://www.apache.org/licenses/LICENSE-2.0
7+
8+
Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on
9+
an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the
10+
specific language governing permissions and limitations under the License.
11+
-->
12+
13+
# Batch inference
14+
15+
Batch inference processes multiple prompts at a time to increase throughput. It is more efficient because processing multiple prompts at once maximizes GPU usage versus processing a single prompt and underutilizing the GPU.
16+
17+
The downside is increased latency because you must wait for the entire batch to complete, and more GPU memory is required for large batches.
18+
19+
To generate a batch of images, pass a list of prompts or images to the pipeline.
20+
21+
<hfoptions id="usage">
22+
<hfoption id="text-to-image">
23+
24+
```py
25+
import torch
26+
from diffusers import DiffusionPipeline
27+
28+
pipeline = DiffusionPipeline.from_pretrained(
29+
"stabilityai/stable-diffusion-xl-base-1.0",
30+
torch_dtype=torch.float16
31+
).to("cuda")
32+
33+
prompts = [
34+
"cinematic photo of A beautiful sunset over mountains, 35mm photograph, film, professional, 4k, highly detailed",
35+
"cinematic film still of a cat basking in the sun on a roof in Turkey, highly detailed, high budget hollywood movie, cinemascope, moody, epic, gorgeous, film grain",
36+
"pixel-art a cozy coffee shop interior, low-res, blocky, pixel art style, 8-bit graphics"
37+
]
38+
39+
images = pipeline(
40+
prompt=prompts,
41+
).images
42+
43+
for i, image in enumerate(images):
44+
image.save(f"batch_image_{i}.png")
45+
print(f"Generated image {i+1} for prompt: {prompts[i]}")
46+
```
47+
48+
</hfoption>
49+
<hfoption id="image-to-image">
50+
51+
```py
52+
import torch
53+
from diffusers.utils import load_image
54+
from diffusers import DiffusionPipeline
55+
56+
pipeline = DiffusionPipeline.from_pretrained(
57+
"stabilityai/stable-diffusion-xl-base-1.0",
58+
torch_dtype=torch.float16
59+
).to("cuda")
60+
61+
input_images = [
62+
load_image("https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/inpaint.png"),
63+
load_image("https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/cat.png"),
64+
load_image("https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/detail-prompt.png")
65+
]
66+
67+
prompts = [
68+
"cinematic photo of a beautiful sunset over mountains, 35mm photograph, film, professional, 4k, highly detailed",
69+
"cinematic film still of a cat basking in the sun on a roof in Turkey, highly detailed, high budget hollywood movie, cinemascope, moody, epic, gorgeous, film grain",
70+
"pixel-art a cozy coffee shop interior, low-res, blocky, pixel art style, 8-bit graphics"
71+
]
72+
73+
images = pipeline(
74+
prompt=prompts,
75+
image=input_images,
76+
guidance_scale=8.0,
77+
strength=0.5
78+
).images
79+
80+
for i, image in enumerate(images):
81+
image.save(f"batch_image_{i}.png")
82+
print(f"Generated image {i+1} for prompt: {prompts[i]}")
83+
```
84+
85+
</hfoption>
86+
</hfoptions>
87+
88+
## Deterministic generation
89+
90+
Enable reproducible batch generation by passing a list of [Generator’s](https://pytorch.org/docs/stable/generated/torch.Generator.html) to the pipeline and tie each `Generator` to a seed to reuse it.
91+
92+
Use a list comprehension to iterate over the batch size specified in `range()` to create a unique `Generator` object for each image in the batch. Don't multiply the `Generator` by the batch size because that only creates one `Generator` object that is used sequentially for each image in the batch.
93+
94+
Pass the `geneator` to the pipeline.
95+
96+
```py
97+
import torch
98+
from diffusers import DiffusionPipeline
99+
100+
pipeline = DiffusionPipeline.from_pretrained(
101+
"stabilityai/stable-diffusion-xl-base-1.0",
102+
torch_dtype=torch.float16
103+
).to("cuda")
104+
105+
generator = [torch.Generator(device="cuda").manual_seed(i) for i in range(3)]
106+
prompts = [
107+
"cinematic photo of A beautiful sunset over mountains, 35mm photograph, film, professional, 4k, highly detailed",
108+
"cinematic film still of a cat basking in the sun on a roof in Turkey, highly detailed, high budget hollywood movie, cinemascope, moody, epic, gorgeous, film grain",
109+
"pixel-art a cozy coffee shop interior, low-res, blocky, pixel art style, 8-bit graphics"
110+
]
111+
112+
images = pipeline(
113+
prompt=prompts,
114+
generator=generator
115+
).images
116+
117+
for i, image in enumerate(images):
118+
image.save(f"batch_image_{i}.png")
119+
print(f"Generated image {i+1} for prompt: {prompts[i]}")
120+
```
121+
122+
You can use this to iteratively select an image associated with a seed and then improve on it by crafting a more detailed prompt.

‎docs/source/en/using-diffusers/reusing_seeds.md‎

Lines changed: 0 additions & 50 deletions
Original file line numberDiff line numberDiff line change
@@ -136,53 +136,3 @@ result2 = pipe(prompt=prompt, num_inference_steps=50, generator=g, output_type="
136136
print("L_inf dist =", abs(result1 - result2).max())
137137
"L_inf dist = tensor(0., device='cuda:0')"
138138
```
139-
140-
## Deterministic batch generation
141-
142-
A practical application of creating reproducible pipelines is *deterministic batch generation*. You generate a batch of images and select one image to improve with a more detailed prompt. The main idea is to pass a list of [Generator's](https://pytorch.org/docs/stable/generated/torch.Generator.html) to the pipeline and tie each `Generator` to a seed so you can reuse it.
143-
144-
Let's use the [stable-diffusion-v1-5/stable-diffusion-v1-5](https://huggingface.co/stable-diffusion-v1-5/stable-diffusion-v1-5) checkpoint and generate a batch of images.
145-
146-
```py
147-
import torch
148-
from diffusers import DiffusionPipeline
149-
from diffusers.utils import make_image_grid
150-
151-
pipeline = DiffusionPipeline.from_pretrained(
152-
"stable-diffusion-v1-5/stable-diffusion-v1-5", torch_dtype=torch.float16, use_safetensors=True
153-
)
154-
pipeline = pipeline.to("cuda")
155-
```
156-
157-
Define four different `Generator`s and assign each `Generator` a seed (`0` to `3`). Then generate a batch of images and pick one to iterate on.
158-
159-
> [!WARNING]
160-
> Use a list comprehension that iterates over the batch size specified in `range()` to create a unique `Generator` object for each image in the batch. If you multiply the `Generator` by the batch size integer, it only creates *one* `Generator` object that is used sequentially for each image in the batch.
161-
>
162-
> ```py
163-
> [torch.Generator().manual_seed(seed)] * 4
164-
> ```
165-
166-
```python
167-
generator = [torch.Generator(device="cuda").manual_seed(i) for i in range(4)]
168-
prompt = "Labrador in the style of Vermeer"
169-
images = pipeline(prompt, generator=generator, num_images_per_prompt=4).images[0]
170-
make_image_grid(images, rows=2, cols=2)
171-
```
172-
173-
<div class="flex justify-center">
174-
<img src="https://huggingface.co/datasets/diffusers/diffusers-images-docs/resolve/main/reusabe_seeds.jpg"/>
175-
</div>
176-
177-
Let's improve the first image (you can choose any image you want) which corresponds to the `Generator` with seed `0`. Add some additional text to your prompt and then make sure you reuse the same `Generator` with seed `0`. All the generated images should resemble the first image.
178-
179-
```python
180-
prompt = [prompt + t for t in [", highly realistic", ", artsy", ", trending", ", colorful"]]
181-
generator = [torch.Generator(device="cuda").manual_seed(0) for i in range(4)]
182-
images = pipeline(prompt, generator=generator).images
183-
make_image_grid(images, rows=2, cols=2)
184-
```
185-
186-
<div class="flex justify-center">
187-
<img src="https://huggingface.co/datasets/diffusers/diffusers-images-docs/resolve/main/reusabe_seeds_2.jpg"/>
188-
</div>

0 commit comments

Comments
 (0)