Skip to content

Commit 4a79ebf

Browse files
stevhliustevhliu
committed
fix toctree
1 parent b933d5b commit 4a79ebf

File tree

2 files changed

+8
-208
lines changed

2 files changed

+8
-208
lines changed

‎docs/source/en/_toctree.yml‎

Lines changed: 0 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -65,8 +65,6 @@
6565
title: Create a server
6666
- local: training/distributed_inference
6767
title: Distributed inference
68-
- local: using-diffusers/merge_loras
69-
title: Merge LoRAs
7068
- local: using-diffusers/scheduler_features
7169
title: Scheduler features
7270
- local: using-diffusers/callback

‎docs/source/en/tutorials/using_peft_for_inference.md‎

Lines changed: 8 additions & 206 deletions
Original file line numberDiff line numberDiff line change
@@ -12,216 +12,18 @@ specific language governing permissions and limitations under the License.
1212

1313
[[open-in-colab]]
1414

15-
# Load LoRAs for inference
15+
# LoRA
1616

17-
There are many adapter types (with [LoRAs](https://huggingface.co/docs/peft/conceptual_guides/adapter#low-rank-adaptation-lora) being the most popular) trained in different styles to achieve different effects. You can even combine multiple adapters to create new and unique images.
17+
## Adjust weight scale
1818

19-
In this tutorial, you'll learn how to easily load and manage adapters for inference with the 🤗 [PEFT](https://huggingface.co/docs/peft/index) integration in 🤗 Diffusers. You'll use LoRA as the main adapter technique, so you'll see the terms LoRA and adapter used interchangeably.
19+
## Hotswap
2020

21-
Let's first install all the required libraries.
21+
## Merge
2222

23-
```bash
24-
!pip install -q transformers accelerate peft diffusers
25-
```
23+
### set_adapters
2624

27-
Now, load a pipeline with a [Stable Diffusion XL (SDXL)](../api/pipelines/stable_diffusion/stable_diffusion_xl) checkpoint:
25+
### add_weighted_adapter
2826

29-
```python
30-
from diffusers import DiffusionPipeline
31-
import torch
27+
### fuse_lora
3228

33-
pipe_id = "stabilityai/stable-diffusion-xl-base-1.0"
34-
pipe = DiffusionPipeline.from_pretrained(pipe_id, torch_dtype=torch.float16).to("cuda")
35-
```
36-
37-
Next, load a [CiroN2022/toy-face](https://huggingface.co/CiroN2022/toy-face) adapter with the [`~diffusers.loaders.StableDiffusionXLLoraLoaderMixin.load_lora_weights`] method. With the 🤗 PEFT integration, you can assign a specific `adapter_name` to the checkpoint, which lets you easily switch between different LoRA checkpoints. Let's call this adapter `"toy"`.
38-
39-
```python
40-
pipe.load_lora_weights("CiroN2022/toy-face", weight_name="toy_face_sdxl.safetensors", adapter_name="toy")
41-
```
42-
43-
Make sure to include the token `toy_face` in the prompt and then you can perform inference:
44-
45-
```python
46-
prompt = "toy_face of a hacker with a hoodie"
47-
48-
lora_scale = 0.9
49-
image = pipe(
50-
prompt, num_inference_steps=30, cross_attention_kwargs={"scale": lora_scale}, generator=torch.manual_seed(0)
51-
).images[0]
52-
image
53-
```
54-
55-
![toy-face](https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/peft_integration/diffusers_peft_lora_inference_8_1.png)
56-
57-
With the `adapter_name` parameter, it is really easy to use another adapter for inference! Load the [nerijs/pixel-art-xl](https://huggingface.co/nerijs/pixel-art-xl) adapter that has been fine-tuned to generate pixel art images and call it `"pixel"`.
58-
59-
The pipeline automatically sets the first loaded adapter (`"toy"`) as the active adapter, but you can activate the `"pixel"` adapter with the [`~loaders.peft.PeftAdapterMixin.set_adapters`] method:
60-
61-
```python
62-
pipe.load_lora_weights("nerijs/pixel-art-xl", weight_name="pixel-art-xl.safetensors", adapter_name="pixel")
63-
pipe.set_adapters("pixel")
64-
```
65-
66-
Make sure you include the token `pixel art` in your prompt to generate a pixel art image:
67-
68-
```python
69-
prompt = "a hacker with a hoodie, pixel art"
70-
image = pipe(
71-
prompt, num_inference_steps=30, cross_attention_kwargs={"scale": lora_scale}, generator=torch.manual_seed(0)
72-
).images[0]
73-
image
74-
```
75-
76-
![pixel-art](https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/peft_integration/diffusers_peft_lora_inference_12_1.png)
77-
78-
<Tip>
79-
80-
By default, if the most up-to-date versions of PEFT and Transformers are detected, `low_cpu_mem_usage` is set to `True` to speed up the loading time of LoRA checkpoints.
81-
82-
</Tip>
83-
84-
## Merge adapters
85-
86-
You can also merge different adapter checkpoints for inference to blend their styles together.
87-
88-
Once again, use the [`~loaders.peft.PeftAdapterMixin.set_adapters`] method to activate the `pixel` and `toy` adapters and specify the weights for how they should be merged.
89-
90-
```python
91-
pipe.set_adapters(["pixel", "toy"], adapter_weights=[0.5, 1.0])
92-
```
93-
94-
<Tip>
95-
96-
LoRA checkpoints in the diffusion community are almost always obtained with [DreamBooth](https://huggingface.co/docs/diffusers/main/en/training/dreambooth). DreamBooth training often relies on "trigger" words in the input text prompts in order for the generation results to look as expected. When you combine multiple LoRA checkpoints, it's important to ensure the trigger words for the corresponding LoRA checkpoints are present in the input text prompts.
97-
98-
</Tip>
99-
100-
Remember to use the trigger words for [CiroN2022/toy-face](https://hf.co/CiroN2022/toy-face) and [nerijs/pixel-art-xl](https://hf.co/nerijs/pixel-art-xl) (these are found in their repositories) in the prompt to generate an image.
101-
102-
```python
103-
prompt = "toy_face of a hacker with a hoodie, pixel art"
104-
image = pipe(
105-
prompt, num_inference_steps=30, cross_attention_kwargs={"scale": 1.0}, generator=torch.manual_seed(0)
106-
).images[0]
107-
image
108-
```
109-
110-
![toy-face-pixel-art](https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/peft_integration/diffusers_peft_lora_inference_16_1.png)
111-
112-
Impressive! As you can see, the model generated an image that mixed the characteristics of both adapters.
113-
114-
> [!TIP]
115-
> Through its PEFT integration, Diffusers also offers more efficient merging methods which you can learn about in the [Merge LoRAs](../using-diffusers/merge_loras) guide!
116-
117-
To return to only using one adapter, use the [`~loaders.peft.PeftAdapterMixin.set_adapters`] method to activate the `"toy"` adapter:
118-
119-
```python
120-
pipe.set_adapters("toy")
121-
122-
prompt = "toy_face of a hacker with a hoodie"
123-
lora_scale = 0.9
124-
image = pipe(
125-
prompt, num_inference_steps=30, cross_attention_kwargs={"scale": lora_scale}, generator=torch.manual_seed(0)
126-
).images[0]
127-
image
128-
```
129-
130-
Or to disable all adapters entirely, use the [`~loaders.peft.PeftAdapterMixin.disable_lora`] method to return the base model.
131-
132-
```python
133-
pipe.disable_lora()
134-
135-
prompt = "toy_face of a hacker with a hoodie"
136-
image = pipe(prompt, num_inference_steps=30, generator=torch.manual_seed(0)).images[0]
137-
image
138-
```
139-
140-
![no-lora](https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/peft_integration/diffusers_peft_lora_inference_20_1.png)
141-
142-
### Customize adapters strength
143-
144-
For even more customization, you can control how strongly the adapter affects each part of the pipeline. For this, pass a dictionary with the control strengths (called "scales") to [`~loaders.peft.PeftAdapterMixin.set_adapters`].
145-
146-
For example, here's how you can turn on the adapter for the `down` parts, but turn it off for the `mid` and `up` parts:
147-
```python
148-
pipe.enable_lora() # enable lora again, after we disabled it above
149-
prompt = "toy_face of a hacker with a hoodie, pixel art"
150-
adapter_weight_scales = { "unet": { "down": 1, "mid": 0, "up": 0} }
151-
pipe.set_adapters("pixel", adapter_weight_scales)
152-
image = pipe(prompt, num_inference_steps=30, generator=torch.manual_seed(0)).images[0]
153-
image
154-
```
155-
156-
![block-lora-text-and-down](https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/peft_integration/diffusers_peft_lora_inference_block_down.png)
157-
158-
Let's see how turning off the `down` part and turning on the `mid` and `up` part respectively changes the image.
159-
```python
160-
adapter_weight_scales = { "unet": { "down": 0, "mid": 1, "up": 0} }
161-
pipe.set_adapters("pixel", adapter_weight_scales)
162-
image = pipe(prompt, num_inference_steps=30, generator=torch.manual_seed(0)).images[0]
163-
image
164-
```
165-
166-
![block-lora-text-and-mid](https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/peft_integration/diffusers_peft_lora_inference_block_mid.png)
167-
168-
```python
169-
adapter_weight_scales = { "unet": { "down": 0, "mid": 0, "up": 1} }
170-
pipe.set_adapters("pixel", adapter_weight_scales)
171-
image = pipe(prompt, num_inference_steps=30, generator=torch.manual_seed(0)).images[0]
172-
image
173-
```
174-
175-
![block-lora-text-and-up](https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/peft_integration/diffusers_peft_lora_inference_block_up.png)
176-
177-
Looks cool!
178-
179-
This is a really powerful feature. You can use it to control the adapter strengths down to per-transformer level. And you can even use it for multiple adapters.
180-
```python
181-
adapter_weight_scales_toy = 0.5
182-
adapter_weight_scales_pixel = {
183-
"unet": {
184-
"down": 0.9, # all transformers in the down-part will use scale 0.9
185-
# "mid" # because, in this example, "mid" is not given, all transformers in the mid part will use the default scale 1.0
186-
"up": {
187-
"block_0": 0.6, # all 3 transformers in the 0th block in the up-part will use scale 0.6
188-
"block_1": [0.4, 0.8, 1.0], # the 3 transformers in the 1st block in the up-part will use scales 0.4, 0.8 and 1.0 respectively
189-
}
190-
}
191-
}
192-
pipe.set_adapters(["toy", "pixel"], [adapter_weight_scales_toy, adapter_weight_scales_pixel])
193-
image = pipe(prompt, num_inference_steps=30, generator=torch.manual_seed(0)).images[0]
194-
image
195-
```
196-
197-
![block-lora-mixed](https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/peft_integration/diffusers_peft_lora_inference_block_mixed.png)
198-
199-
## Manage adapters
200-
201-
You have attached multiple adapters in this tutorial, and if you're feeling a bit lost on what adapters have been attached to the pipeline's components, use the [`~diffusers.loaders.StableDiffusionLoraLoaderMixin.get_active_adapters`] method to check the list of active adapters:
202-
203-
```py
204-
active_adapters = pipe.get_active_adapters()
205-
active_adapters
206-
["toy", "pixel"]
207-
```
208-
209-
You can also get the active adapters of each pipeline component with [`~diffusers.loaders.StableDiffusionLoraLoaderMixin.get_list_adapters`]:
210-
211-
```py
212-
list_adapters_component_wise = pipe.get_list_adapters()
213-
list_adapters_component_wise
214-
{"text_encoder": ["toy", "pixel"], "unet": ["toy", "pixel"], "text_encoder_2": ["toy", "pixel"]}
215-
```
216-
217-
The [`~loaders.peft.PeftAdapterMixin.delete_adapters`] function completely removes an adapter and their LoRA layers from a model.
218-
219-
```py
220-
pipe.delete_adapters("toy")
221-
pipe.get_active_adapters()
222-
["pixel"]
223-
```
224-
225-
## PeftInputAutocastDisableHook
226-
227-
[[autodoc]] hooks.layerwise_casting.PeftInputAutocastDisableHook
29+
## torch.compile

0 commit comments

Comments
 (0)