You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
To use bitsandbytes, make sure you have the following libraries installed:
22
28
@@ -31,70 +37,167 @@ Now you can quantize a model by passing a [`BitsAndBytesConfig`] to [`~ModelMixi
31
37
32
38
Quantizing a model in 8-bit halves the memory-usage:
33
39
40
+
bitsandbytes is supported in both Transformers and Diffusers, so you can quantize both the
41
+
[`FluxTransformer2DModel`] and [`~transformers.T5EncoderModel`].
42
+
43
+
For Ada and higher-series GPUs. we recommend changing `torch_dtype` to `torch.bfloat16`.
44
+
45
+
> [!TIP]
46
+
> The [`CLIPTextModel`] and [`AutoencoderKL`] aren't quantized because they're already small in size and because [`AutoencoderKL`] only has a few `torch.nn.Linear` layers.
47
+
34
48
```py
35
-
from diffusers import FluxTransformer2DModel, BitsAndBytesConfig
49
+
from diffusers import BitsAndBytesConfig as DiffusersBitsAndBytesConfig
50
+
from transformers import BitsAndBytesConfig as TransformersBitsAndBytesConfig
By default, all the other modules such as `torch.nn.LayerNorm` are converted to `torch.float16`. You can change the data type of these modules with the `torch_dtype` parameter if you want:
74
+
By default, all the other modules such as `torch.nn.LayerNorm` are converted to `torch.float16`. You can change the data type of these modules with the `torch_dtype` parameter.
47
75
48
-
```py
49
-
from diffusers import FluxTransformer2DModel, BitsAndBytesConfig
Once a model is quantized, you can push the model to the Hub with the [`~ModelMixin.push_to_hub`] method. The quantization `config.json` file is pushed first, followed by the quantized model weights. You can also save the serialized 4-bit models locally with [`~ModelMixin.save_pretrained`].
When there is enough memory, you can also directly move the pipeline to the GPU with `.to("cuda")` and apply [`~DiffusionPipeline.enable_model_cpu_offload`] to optimize GPU memory usage.
116
+
117
+
Once a model is quantized, you can push the model to the Hub with the [`~ModelMixin.push_to_hub`] method. The quantization `config.json` file is pushed first, followed by the quantized model weights. You can also save the serialized 8-bit models locally with [`~ModelMixin.save_pretrained`].
63
118
64
119
</hfoption>
65
120
<hfoptionid="4-bit">
66
121
67
122
Quantizing a model in 4-bit reduces your memory-usage by 4x:
68
123
124
+
bitsandbytes is supported in both Transformers and Diffusers, so you can can quantize both the
125
+
[`FluxTransformer2DModel`] and [`~transformers.T5EncoderModel`].
126
+
127
+
For Ada and higher-series GPUs. we recommend changing `torch_dtype` to `torch.bfloat16`.
128
+
129
+
> [!TIP]
130
+
> The [`CLIPTextModel`] and [`AutoencoderKL`] aren't quantized because they're already small in size and because [`AutoencoderKL`] only has a few `torch.nn.Linear` layers.
131
+
69
132
```py
70
-
from diffusers import FluxTransformer2DModel, BitsAndBytesConfig
133
+
from diffusers import BitsAndBytesConfig as DiffusersBitsAndBytesConfig
134
+
from transformers import BitsAndBytesConfig as TransformersBitsAndBytesConfig
By default, all the other modules such as `torch.nn.LayerNorm` are converted to `torch.float16`. You can change the data type of these modules with the `torch_dtype` parameter if you want:
158
+
By default, all the other modules such as `torch.nn.LayerNorm` are converted to `torch.float16`. You can change the data type of these modules with the `torch_dtype` parameter.
82
159
83
-
```py
84
-
from diffusers import FluxTransformer2DModel, BitsAndBytesConfig
Setting `device_map="auto"` automatically fills all available space on the GPU(s) first, then the CPU, and finally, the hard drive (the absolute slowest option) if there is still not enough memory.
Call [`~ModelMixin.push_to_hub`] after loading it in 4-bit precision. You can also save the serialized 4-bit models locally with [`~ModelMixin.save_pretrained`].
When there is enough memory, you can also directly move the pipeline to the GPU with `.to("cuda")` and apply [`~DiffusionPipeline.enable_model_cpu_offload`] to optimize GPU memory usage.
199
+
200
+
Once a model is quantized, you can push the model to the Hub with the [`~ModelMixin.push_to_hub`] method. The quantization `config.json` file is pushed first, followed by the quantized model weights. You can also save the serialized 4-bit models locally with [`~ModelMixin.save_pretrained`].
NF4 is a 4-bit data type from the [QLoRA](https://hf.co/papers/2305.14314) paper, adapted for weights initialized from a normal distribution. You should use NF4 for training 4-bit base models. This can be configured with the `bnb_4bit_quant_type` parameter in the [`BitsAndBytesConfig`]:
200
303
201
304
```py
202
-
from diffusers import BitsAndBytesConfig
305
+
from diffusers import BitsAndBytesConfig as DiffusersBitsAndBytesConfig
306
+
from transformers import BitsAndBytesConfig as TransformersBitsAndBytesConfig
@@ -220,38 +340,74 @@ For inference, the `bnb_4bit_quant_type` does not have a huge impact on performa
220
340
Nested quantization is a technique that can save additional memory at no additional performance cost. This feature performs a second quantization of the already quantized weights to save an additional 0.4 bits/parameter.
221
341
222
342
```py
223
-
from diffusers import BitsAndBytesConfig
343
+
from diffusers import BitsAndBytesConfig as DiffusersBitsAndBytesConfig
344
+
from transformers import BitsAndBytesConfig as TransformersBitsAndBytesConfig
Once quantized, you can dequantize the model to the original precision but this might result in a small quality loss of the model. Make sure you have enough GPU RAM to fit the dequantized model.
376
+
Once quantized, you can dequantize a model to its original precision, but this might result in a small loss of quality. Make sure you have enough GPU RAM to fit the dequantized model.
240
377
241
378
```python
242
-
from diffusers import BitsAndBytesConfig
379
+
from diffusers import BitsAndBytesConfig as DiffusersBitsAndBytesConfig
380
+
from transformers import BitsAndBytesConfig as TransformersBitsAndBytesConfig
Copy file name to clipboardExpand all lines: docs/source/en/using-diffusers/loading_adapters.md
+6-2Lines changed: 6 additions & 2 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -134,14 +134,16 @@ The [`~loaders.StableDiffusionLoraLoaderMixin.load_lora_weights`] method loads L
134
134
- the LoRA weights don't have separate identifiers for the UNet and text encoder
135
135
- the LoRA weights have separate identifiers for the UNet and text encoder
136
136
137
-
But if you only need to load LoRA weights into the UNet, then you can use the [`~loaders.UNet2DConditionLoadersMixin.load_attn_procs`] method. Let's load the [jbilcke-hf/sdxl-cinematic-1](https://huggingface.co/jbilcke-hf/sdxl-cinematic-1) LoRA:
137
+
To directly load (and save) a LoRA adapter at the *model-level*, use [`~PeftAdapterMixin.load_lora_adapter`], which builds and prepares the necessary model configuration for the adapter. Like [`~loaders.StableDiffusionLoraLoaderMixin.load_lora_weights`], [`PeftAdapterMixin.load_lora_adapter`] can load LoRAs for both the UNet and text encoder. For example, if you're loading a LoRA for the UNet, [`PeftAdapterMixin.load_lora_adapter`] ignores the keys for the text encoder.
138
+
139
+
Use the `weight_name` parameter to specify the specific weight file and the `prefix` parameter to filter for the appropriate state dicts (`"unet"` in this case) to load.
Save an adapter with [`~PeftAdapterMixin.save_lora_adapter`].
159
+
156
160
To unload the LoRA weights, use the [`~loaders.StableDiffusionLoraLoaderMixin.unload_lora_weights`] method to discard the LoRA weights and restore the model to its original weights:
0 commit comments