Skip to content

Commit 4804156

Browse files
vuongminh1907vuongminh1907
authored
Merge branch 'main' into main
2 parents 3ef54fa + bb1d9a8 commit 4804156

27 files changed

+4712
-3
lines changed

‎docs/source/en/quicktour.md‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -162,6 +162,9 @@ Take a look at the [Quantization](./quantization/overview) section for more deta
162162

163163
## Optimizations
164164

165+
> [!TIP]
166+
> Optimization is dependent on hardware specs such as memory. Use this [Space](https://huggingface.co/spaces/diffusers/optimized-diffusers-code) to generate code examples that include all of Diffusers' available memory and speed optimization techniques for any model you're using.
167+
165168
Modern diffusion models are very large and have billions of parameters. The iterative denoising process is also computationally intensive and slow. Diffusers provides techniques for reducing memory usage and boosting inference speed. These techniques can be combined with quantization to optimize for both memory usage and inference speed.
166169

167170
### Memory usage

‎docs/source/zh/_toctree.yml‎

Lines changed: 56 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -15,15 +15,49 @@
1515
- local: using-diffusers/schedulers
1616
title: Load schedulers and models
1717

18+
- title: Inference
19+
isExpanded: false
20+
sections:
21+
- local: training/distributed_inference
22+
title: Distributed inference
23+
1824
- title: Inference optimization
1925
isExpanded: false
2026
sections:
2127
- local: optimization/fp16
2228
title: Accelerate inference
29+
- local: optimization/cache
30+
title: Caching
31+
- local: optimization/memory
32+
title: Reduce memory usage
33+
- local: optimization/speed-memory-optims
34+
title: Compile and offloading quantized models
2335
- title: Community optimizations
2436
sections:
37+
- local: optimization/pruna
38+
title: Pruna
2539
- local: optimization/xformers
2640
title: xFormers
41+
- local: optimization/tome
42+
title: Token merging
43+
- local: optimization/deepcache
44+
title: DeepCache
45+
- local: optimization/tgate
46+
title: TGATE
47+
- local: optimization/xdit
48+
title: xDiT
49+
- local: optimization/para_attn
50+
title: ParaAttention
51+
52+
- title: Hybrid Inference
53+
isExpanded: false
54+
sections:
55+
- local: hybrid_inference/overview
56+
title: Overview
57+
- local: hybrid_inference/vae_encode
58+
title: VAE Encode
59+
- local: hybrid_inference/api_reference
60+
title: API Reference
2761

2862
- title: Modular Diffusers
2963
isExpanded: false
@@ -44,6 +78,10 @@
4478
title: AutoPipelineBlocks
4579
- local: modular_diffusers/modular_pipeline
4680
title: ModularPipeline
81+
- local: modular_diffusers/components_manager
82+
title: ComponentsManager
83+
- local: modular_diffusers/guiders
84+
title: Guiders
4785

4886
- title: Training
4987
isExpanded: false
@@ -56,12 +94,20 @@
5694
sections:
5795
- local: training/text2image
5896
title: Text-to-image
97+
- local: training/kandinsky
98+
title: Kandinsky 2.2
99+
- local: training/wuerstchen
100+
title: Wuerstchen
59101
- local: training/controlnet
60102
title: ControlNet
103+
- local: training/instructpix2pix
104+
title: InstructPix2Pix
61105
- title: Methods
62106
sections:
63107
- local: training/text_inversion
64108
title: Textual Inversion
109+
- local: training/dreambooth
110+
title: DreamBooth
65111
- local: training/lora
66112
title: LoRA
67113

@@ -70,6 +116,16 @@
70116
sections:
71117
- local: optimization/onnx
72118
title: ONNX
119+
- local: optimization/open_vino
120+
title: OpenVINO
121+
- local: optimization/coreml
122+
title: Core ML
123+
- local: optimization/mps
124+
title: Metal Performance Shaders (MPS)
125+
- local: optimization/habana
126+
title: Intel Gaudi
127+
- local: optimization/neuron
128+
title: AWS Neuron
73129

74130
- title: Specific pipeline examples
75131
isExpanded: false

‎docs/source/zh/hybrid_inference/api_reference.md‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,9 @@
1+
# 混合推理 API 参考
2+
3+
## 远程解码
4+
5+
[[autodoc]] utils.remote_utils.remote_decode
6+
7+
## 远程编码
8+
9+
[[autodoc]] utils.remote_utils.remote_encode

‎docs/source/zh/hybrid_inference/overview.md‎

Lines changed: 55 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,55 @@
1+
<!--版权 2025 HuggingFace 团队。保留所有权利。
2+
3+
根据 Apache 许可证 2.0 版本（"许可证"）授权；除非遵守许可证，否则不得使用此文件。
4+
您可以在以下网址获取许可证副本：
5+
6+
http://www.apache.org/licenses/LICENSE-2.0
7+
8+
除非适用法律要求或书面同意，否则根据许可证分发的软件按"原样"分发，不附带任何明示或暗示的担保或条件。请参阅许可证以了解具体的语言管理权限和限制。
9+
-->
10+
11+
# 混合推理
12+
13+
**通过混合推理赋能本地 AI 构建者**
14+
15+
> [!TIP]
16+
> 混合推理是一项[实验性功能](https://huggingface.co/blog/remote_vae)。
17+
> 可以在此处提供反馈[此处](https://github.com/huggingface/diffusers/issues/new?template=remote-vae-pilot-feedback.yml)。
18+
19+
## 为什么使用混合推理？
20+
21+
混合推理提供了一种快速简单的方式来卸载本地生成需求。
22+
23+
- 🚀 **降低要求：** 无需昂贵硬件即可访问强大模型。
24+
- 💎 **无妥协：** 在不牺牲性能的情况下实现最高质量。
25+
- 💰 **成本效益高：** 它是免费的！🤑
26+
- 🎯 **多样化用例：** 与 Diffusers � 和更广泛的社区完全兼容。
27+
- 🔧 **开发者友好：** 简单请求，快速响应。
28+
29+
---
30+
31+
## 可用模型
32+
33+
* **VAE 解码 🖼️：** 快速将潜在表示解码为高质量图像，不影响性能或工作流速度。
34+
* **VAE 编码 🔢：** 高效将图像编码为潜在表示，用于生成和训练。
35+
* **文本编码器 📃（即将推出）：** 快速准确地计算提示的文本嵌入，确保流畅高质量的工作流。
36+
37+
---
38+
39+
## 集成
40+
41+
* **[SD.Next](https://github.com/vladmandic/sdnext)：** 一体化 UI，直接支持混合推理。
42+
* **[ComfyUI-HFRemoteVae](https://github.com/kijai/ComfyUI-HFRemoteVae)：** 用于混合推理的 ComfyUI 节点。
43+
44+
## 更新日志
45+
46+
- 2025 年 3 月 10 日：添加了 VAE 编码
47+
- 2025 年 3 月 2 日：初始发布，包含 VAE 解码
48+
49+
## 内容
50+
51+
文档分为三个部分：
52+
53+
* **VAE 解码** 学习如何使用混合推理进行 VAE 解码的基础知识。
54+
* **VAE 编码** 学习如何使用混合推理进行 VAE 编码的基础知识。
55+
* **API 参考** 深入了解任务特定设置和参数。

‎docs/source/zh/hybrid_inference/vae_encode.md‎

Lines changed: 184 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,184 @@
1+
# 入门：使用混合推理进行 VAE 编码
2+
3+
VAE 编码用于训练、图像到图像和图像到视频——将图像或视频转换为潜在表示。
4+
5+
## 内存
6+
7+
这些表格展示了在不同 GPU 上使用 SD v1 和 SD XL 进行 VAE 编码的 VRAM 需求。
8+
9+
对于这些 GPU 中的大多数，内存使用百分比决定了其他模型（文本编码器、UNet/Transformer）必须被卸载，或者必须使用分块编码，这会增加时间并影响质量。
10+
11+
<details><summary>SD v1.5</summary>
12+
13+
| GPU | 分辨率 | 时间（秒） | 内存（%） | 分块时间（秒） | 分块内存（%） |
14+
|:------------------------------|:-------------|-----------------:|-------------:|--------------------:|-------------------:|
15+
| NVIDIA GeForce RTX 4090 | 512x512 | 0.015 | 3.51901 | 0.015 | 3.51901 |
16+
| NVIDIA GeForce RTX 4090 | 256x256 | 0.004 | 1.3154 | 0.005 | 1.3154 |
17+
| NVIDIA GeForce RTX 4090 | 2048x2048 | 0.402 | 47.1852 | 0.496 | 3.51901 |
18+
| NVIDIA GeForce RTX 4090 | 1024x1024 | 0.078 | 12.2658 | 0.094 | 3.51901 |
19+
| NVIDIA GeForce RTX 4080 SUPER | 512x512 | 0.023 | 5.30105 | 0.023 | 5.30105 |
20+
| NVIDIA GeForce RTX 4080 SUPER | 256x256 | 0.006 | 1.98152 | 0.006 | 1.98152 |
21+
| NVIDIA GeForce RTX 4080 SUPER | 2048x2048 | 0.574 | 71.08 | 0.656 | 5.30105 |
22+
| NVIDIA GeForce RTX 4080 SUPER | 1024x1024 | 0.111 | 18.4772 | 0.14 | 5.30105 |
23+
| NVIDIA GeForce RTX 3090 | 512x512 | 0.032 | 3.52782 | 0.032 | 3.52782 |
24+
| NVIDIA GeForce RTX 3090 | 256x256 | 0.01 | 1.31869 | 0.009 | 1.31869 |
25+
| NVIDIA GeForce RTX 3090 | 2048x2048 | 0.742 | 47.3033 | 0.954 | 3.52782 |
26+
| NVIDIA GeForce RTX 3090 | 1024x1024 | 0.136 | 12.2965 | 0.207 | 3.52782 |
27+
| NVIDIA GeForce RTX 3080 | 512x512 | 0.036 | 8.51761 | 0.036 | 8.51761 |
28+
| NVIDIA GeForce RTX 3080 | 256x256 | 0.01 | 3.18387 | 0.01 | 3.18387 |
29+
| NVIDIA GeForce RTX 3080 | 2048x2048 | 0.863 | 86.7424 | 1.191 | 8.51761 |
30+
| NVIDIA GeForce RTX 3080 | 1024x1024 | 0.157 | 29.6888 | 0.227 | 8.51761 |
31+
| NVIDIA GeForce RTX 3070 | 512x512 | 0.051 | 10.6941 | 0.051 | 10.6941 |
32+
| NVIDIA GeForce RTX 3070 | 256x256 | 0.015 |
33+
| 3.99743 | 0.015 | 3.99743 |
34+
| NVIDIA GeForce RTX 3070 | 2048x2048 | 1.217 | 96.054 | 1.482 | 10.6941 |
35+
| NVIDIA GeForce RTX 3070 | 1024x1024 | 0.223 | 37.2751 | 0.327 | 10.6941 |
36+
37+
</details>
38+
39+
<details><summary>SDXL</summary>
40+
41+
| GPU | Resolution | Time (seconds) | Memory Consumed (%) | Tiled Time (seconds) | Tiled Memory (%) |
42+
|:------------------------------|:-------------|-----------------:|----------------------:|-----------------------:|-------------------:|
43+
| NVIDIA GeForce RTX 4090 | 512x512 | 0.029 | 4.95707 | 0.029 | 4.95707 |
44+
| NVIDIA GeForce RTX 4090 | 256x256 | 0.007 | 2.29666 | 0.007 | 2.29666 |
45+
| NVIDIA GeForce RTX 4090 | 2048x2048 | 0.873 | 66.3452 | 0.863 | 15.5649 |
46+
| NVIDIA GeForce RTX 4090 | 1024x1024 | 0.142 | 15.5479 | 0.143 | 15.5479 |
47+
| NVIDIA GeForce RTX 4080 SUPER | 512x512 | 0.044 | 7.46735 | 0.044 | 7.46735 |
48+
| NVIDIA GeForce RTX 4080 SUPER | 256x256 | 0.01 | 3.4597 | 0.01 | 3.4597 |
49+
| NVIDIA GeForce RTX 4080 SUPER | 2048x2048 | 1.317 | 87.1615 | 1.291 | 23.447 |
50+
| NVIDIA GeForce RTX 4080 SUPER | 1024x1024 | 0.213 | 23.4215 | 0.214 | 23.4215 |
51+
| NVIDIA GeForce RTX 3090 | 512x512 | 0.058 | 5.65638 | 0.058 | 5.65638 |
52+
| NVIDIA GeForce RTX 3090 | 256x256 | 0.016 | 2.45081 | 0.016 | 2.45081 |
53+
| NVIDIA GeForce RTX 3090 | 2048x2048 | 1.755 | 77.8239 | 1.614 | 18.4193 |
54+
| NVIDIA GeForce RTX 3090 | 1024x1024 | 0.265 | 18.4023 | 0.265 | 18.4023 |
55+
| NVIDIA GeForce RTX 3080 | 512x512 | 0.064 | 13.6568 | 0.064 | 13.6568 |
56+
| NVIDIA GeForce RTX 3080 | 256x256 | 0.018 | 5.91728 | 0.018 | 5.91728 |
57+
| NVIDIA GeForce RTX 3080 | 2048x2048 | 内存不足 (OOM) | 内存不足 (OOM) | 1.866 | 44.4717 |
58+
| NVIDIA GeForce RTX 3080 | 1024x1024 | 0.302 | 44.4308 | 0.302 | 44.4308 |
59+
| NVIDIA GeForce RTX 3070 | 512x512 | 0.093 | 17.1465 | 0.093 | 17.1465 |
60+
| NVIDIA GeForce R
61+
| NVIDIA GeForce RTX 3070 | 256x256 | 0.025 | 7.42931 | 0.026 | 7.42931 |
62+
| NVIDIA GeForce RTX 3070 | 2048x2048 | OOM | OOM | 2.674 | 55.8355 |
63+
| NVIDIA GeForce RTX 3070 | 1024x1024 | 0.443 | 55.7841 | 0.443 | 55.7841 |
64+
65+
</details>
66+
67+
## 可用 VAE
68+
69+
| | **端点** | **模型** |
70+
|:-:|:-----------:|:--------:|
71+
| **Stable Diffusion v1** | [https://qc6479g0aac6qwy9.us-east-1.aws.endpoints.huggingface.cloud](https://qc6479g0aac6qwy9.us-east-1.aws.endpoints.huggingface.cloud) | [`stabilityai/sd-vae-ft-mse`](https://hf.co/stabilityai/sd-vae-ft-mse) |
72+
| **Stable Diffusion XL** | [https://xjqqhmyn62rog84g.us-east-1.aws.endpoints.huggingface.cloud](https://xjqqhmyn62rog84g.us-east-1.aws.endpoints.huggingface.cloud) | [`madebyollin/sdxl-vae-fp16-fix`](https://hf.co/madebyollin/sdxl-vae-fp16-fix) |
73+
| **Flux** | [https://ptccx55jz97f9zgo.us-east-1.aws.endpoints.huggingface.cloud](https://ptccx55jz97f9zgo.us-east-1.aws.endpoints.huggingface.cloud) | [`black-forest-labs/FLUX.1-schnell`](https://hf.co/black-forest-labs/FLUX.1-schnell) |
74+
75+
76+
> [!TIP]
77+
> 模型支持可以在此处请求：[这里](https://github.com/huggingface/diffusers/issues/new?template=remote-vae-pilot-feedback.yml)。
78+
79+
80+
## 代码
81+
82+
> [!TIP]
83+
> 从 `main` 安装 `diffusers` 以运行代码：`pip install git+https://github.com/huggingface/diffusers@main`
84+
85+
86+
一个辅助方法简化了与混合推理的交互。
87+
88+
```python
89+
from diffusers.utils.remote_utils import remote_encode
90+
```
91+
92+
### 基本示例
93+
94+
让我们编码一张图像，然后解码以演示。
95+
96+
<figure class="image flex flex-col items-center justify-center text-center m-0 w-full">
97+
<img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/astronaut.jpg"/>
98+
</figure>
99+
100+
<details><summary>代码</summary>
101+
102+
```python
103+
from diffusers.utils import load_image
104+
from diffusers.utils.remote_utils import remote_decode
105+
106+
image = load_image("https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/astronaut.jpg?download=true")
107+
108+
latent = remote_encode(
109+
endpoint="https://ptccx55jz97f9zgo.us-east-1.aws.endpoints.huggingface.cloud/",
110+
scaling_factor=0.3611,
111+
shift_factor=0.1159,
112+
)
113+
114+
decoded = remote_decode(
115+
endpoint="https://whhx50ex1aryqvw6.us-east-1.aws.endpoints.huggingface.cloud/",
116+
tensor=latent,
117+
scaling_factor=0.3611,
118+
shift_factor=0.1159,
119+
)
120+
```
121+
122+
</details>
123+
124+
<figure class="image flex flex-col items-center justify-center text-center m-0 w-full">
125+
<img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/blog/remote_vae/decoded.png"/>
126+
</figure>
127+
128+
129+
### 生成
130+
131+
现在让我们看一个生成示例，我们将编码图像，生成，然后远程解码！
132+
133+
<details><summary>代码</summary>
134+
135+
```python
136+
import torch
137+
from diffusers import StableDiffusionImg2ImgPip
138+
from diffusers.utils import load_image
139+
from diffusers.utils.remote_utils import remote_decode, remote_encode
140+
141+
pipe = StableDiffusionImg2ImgPipeline.from_pretrained(
142+
"stable-diffusion-v1-5/stable-diffusion-v1-5",
143+
torch_dtype=torch.float16,
144+
variant="fp16",
145+
vae=None,
146+
).to("cuda")
147+
148+
init_image = load_image(
149+
"https://raw.githubusercontent.com/CompVis/stable-diffusion/main/assets/stable-samples/img2img/sketch-mountains-input.jpg"
150+
)
151+
init_image = init_image.resize((768, 512))
152+
153+
init_latent = remote_encode(
154+
endpoint="https://qc6479g0aac6qwy9.us-east-1.aws.endpoints.huggingface.cloud/",
155+
image=init_image,
156+
scaling_factor=0.18215,
157+
)
158+
159+
prompt = "A fantasy landscape, trending on artstation"
160+
latent = pipe(
161+
prompt=prompt,
162+
image=init_latent,
163+
strength=0.75,
164+
output_type="latent",
165+
).images
166+
167+
image = remote_decode(
168+
endpoint="https://q1bj3bpq6kzilnsu.us-east-1.aws.endpoints.huggingface.cloud/",
169+
tensor=latent,
170+
scaling_factor=0.18215,
171+
)
172+
image.save("fantasy_landscape.jpg")
173+
```
174+
175+
</details>
176+
177+
<figure class="image flex flex-col items-center justify-center text-center m-0 w-full">
178+
<img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/blog/remote_vae/fantasy_landscape.png"/>
179+
</figure>
180+
181+
## 集成
182+
183+
* **[SD.Next](https://github.com/vladmandic/sdnext):** 具有直接支持混合推理功能的一体化用户界面。
184+
* **[ComfyUI-HFRemoteVae](https://github.com/kijai/ComfyUI-HFRemoteVae):** 用于混合推理的 ComfyUI 节点。

0 commit comments

Comments
 (0)