Skip to content

Commit 5419877

Browse files
sayakpaulLinoy Tsabanlinoytsaban
authored
add training scripts (#16)
* add training scripts Co-authored-by: Linoy Tsaban <[email protected]> * model cpu offload in validation. * add flux.2 readme * add img2img and tests * cpu offload in log validation * Apply suggestions from code review * fix * up * fixes * remove i2i training tests for now. --------- Co-authored-by: Linoy Tsaban <[email protected]> Co-authored-by: linoytsaban <[email protected]>
1 parent 823f4c3 commit 5419877

File tree

4 files changed

+4322
-0
lines changed

4 files changed

+4322
-0
lines changed

‎examples/dreambooth/README_flux2.md‎

Lines changed: 315 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,315 @@
1+
# DreamBooth training example for FLUX.2 [dev]
2+
3+
[DreamBooth](https://huggingface.co/papers/2208.12242) is a method to personalize image generation models given just a few (3~5) images of a subject/concept.
4+
5+
The `train_dreambooth_lora_flux2.py` script shows how to implement the training procedure for [LoRAs](https://huggingface.co/blog/lora) and adapt it for [FLUX.2 [dev]](https://github.com/black-forest-labs/flux2-dev).
6+
7+
> [!NOTE]
8+
> **Memory consumption**
9+
>
10+
> Flux can be quite expensive to run on consumer hardware devices and as a result finetuning it comes with high memory requirements -
11+
> a LoRA with a rank of 16 can exceed XXGB of VRAM for training. below we provide some tips and tricks to reduce memory consumption during training.
12+
13+
> For more tips & guidance on training on a resource-constrained device and general good practices please check out these great guides and trainers for FLUX:
14+
> 1) [`@bghira`'s guide](https://github.com/bghira/SimpleTuner/blob/main/documentation/quickstart/FLUX2.md)
15+
> 2) [`ostris`'s guide](https://github.com/ostris/ai-toolkit?tab=readme-ov-file#flux2-training)
16+
17+
> [!NOTE]
18+
> **Gated model**
19+
>
20+
> As the model is gated, before using it with diffusers you first need to go to the [FLUX.2 [dev] Hugging Face page](https://huggingface.co/black-forest-labs/FLUX.2-dev), fill in the form and accept the gate. Once you are in, you need to log in so that your system knows you’ve accepted the gate. Use the command below to log in:
21+
22+
```bash
23+
hf auth login
24+
```
25+
26+
This will also allow us to push the trained model parameters to the Hugging Face Hub platform.
27+
28+
## Running locally with PyTorch
29+
30+
### Installing the dependencies
31+
32+
Before running the scripts, make sure to install the library's training dependencies:
33+
34+
**Important**
35+
36+
To make sure you can successfully run the latest versions of the example scripts, we highly recommend **installing from source** and keeping the install up to date as we update the example scripts frequently and install some example-specific requirements. To do this, execute the following steps in a new virtual environment:
37+
38+
```bash
39+
git clone https://github.com/huggingface/diffusers
40+
cd diffusers
41+
pip install -e .
42+
```
43+
44+
Then cd in the `examples/dreambooth` folder and run
45+
```bash
46+
pip install -r requirements_flux.txt
47+
```
48+
49+
And initialize an [🤗Accelerate](https://github.com/huggingface/accelerate/) environment with:
50+
51+
```bash
52+
accelerate config
53+
```
54+
55+
Or for a default accelerate configuration without answering questions about your environment
56+
57+
```bash
58+
accelerate config default
59+
```
60+
61+
Or if your environment doesn't support an interactive shell (e.g., a notebook)
62+
63+
```python
64+
from accelerate.utils import write_basic_config
65+
write_basic_config()
66+
```
67+
68+
When running `accelerate config`, if we specify torch compile mode to True there can be dramatic speedups.
69+
Note also that we use PEFT library as backend for LoRA training, make sure to have `peft>=0.6.0` installed in your environment.
70+
71+
72+
### Dog toy example
73+
74+
Now let's get our dataset. For this example we will use some dog images: https://huggingface.co/datasets/diffusers/dog-example.
75+
76+
Let's first download it locally:
77+
78+
```python
79+
from huggingface_hub import snapshot_download
80+
81+
local_dir = "./dog"
82+
snapshot_download(
83+
"diffusers/dog-example",
84+
local_dir=local_dir, repo_type="dataset",
85+
ignore_patterns=".gitattributes",
86+
)
87+
```
88+
89+
This will also allow us to push the trained LoRA parameters to the Hugging Face Hub platform.
90+
91+
As mentioned, Flux2 LoRA training is *very* memory intensive. Here are memory optimizations we can use (some still experimental) for a more memory efficient training:
92+
93+
## Memory Optimizations
94+
> [!NOTE] many of these techniques complement each other and can be used together to further reduce memory consumption.
95+
> However some techniques may be mutually exclusive so be sure to check before launching a training run.
96+
### Remote Text Encoder
97+
Flux.2 uses Mistral Small 3.1 as text encoder which is quite large and can take up a lot of memory. To mitigate this, we can use the `--remote_text_encoder` flag to enable remote computation of the prompt embeddings using the HuggingFace Inference API.
98+
This way, the text encoder model is not loaded into memory during training.
99+
> [!NOTE]
100+
> to enable remote text encoding you must either be logged in to your HuggingFace account (`hf auth login`) OR pass a token with `--hub_token`.
101+
### CPU Offloading
102+
To offload parts of the model to CPU memory, you can use `--offload` flag. This will offload the vae and text encoder to CPU memory and only move them to GPU when needed.
103+
### Latent Caching
104+
Pre-encode the training images with the vae, and then delete it to free up some memory. To enable `latent_caching` simply pass `--cache_latents`.
105+
### QLoRA: Low Precision Training with Quantization
106+
Perform low precision training using 8-bit or 4-bit quantization to reduce memory usage. You can use the following flags:
107+
- **FP8 training** with `torchao`:
108+
enable FP8 training by passing `--do_fp8_training`.
109+
> [!IMPORTANT] Since we are utilizing FP8 tensor cores we need CUDA GPUs with compute capability at least 8.9 or greater.
110+
> If you're looking for memory-efficient training on relatively older cards, we encourage you to check out other trainers like SimpleTuner, ai-toolkit, etc.
111+
- **NF4 training** with `bitsandbytes`:
112+
Alternatively, you can use 8-bit or 4-bit quantization with `bitsandbytes` by passing:
113+
`--bnb_quantization_config_path` to enable 4-bit NF4 quantization.
114+
### Gradient Checkpointing and Accumulation
115+
* `--gradient accumulation` refers to the number of updates steps to accumulate before performing a backward/update pass.
116+
by passing a value > 1 you can reduce the amount of backward/update passes and hence also memory reqs.
117+
* with `--gradient checkpointing` we can save memory by not storing all intermediate activations during the forward pass.
118+
Instead, only a subset of these activations (the checkpoints) are stored and the rest is recomputed as needed during the backward pass. Note that this comes at the expanse of a slower backward pass.
119+
### 8-bit-Adam Optimizer
120+
When training with `AdamW`(doesn't apply to `prodigy`) You can pass `--use_8bit_adam` to reduce the memory requirements of training.
121+
Make sure to install `bitsandbytes` if you want to do so.
122+
### Image Resolution
123+
An easy way to mitigate some of the memory requirements is through `--resolution`. `--resolution` refers to the resolution for input images, all the images in the train/validation dataset are resized to this.
124+
Note that by default, images are resized to resolution of 512, but it's good to keep in mind in case you're accustomed to training on higher resolutions.
125+
### Precision of saved LoRA layers
126+
By default, trained transformer layers are saved in the precision dtype in which training was performed. E.g. when training in mixed precision is enabled with `--mixed_precision="bf16"`, final finetuned layers will be saved in `torch.bfloat16` as well.
127+
This reduces memory requirements significantly w/o a significant quality loss. Note that if you do wish to save the final layers in float32 at the expanse of more memory usage, you can do so by passing `--upcast_before_saving`.
128+
129+
130+
```bash
131+
export MODEL_NAME="black-forest-labs/FLUX.2-dev"
132+
export INSTANCE_DIR="dog"
133+
export OUTPUT_DIR="trained-flux2"
134+
135+
accelerate launch train_dreambooth_flux.py \
136+
--pretrained_model_name_or_path=$MODEL_NAME \
137+
--instance_data_dir=$INSTANCE_DIR \
138+
--output_dir=$OUTPUT_DIR \
139+
--do_fp8_training \
140+
--gradient_checkpointing \
141+
--remote_text_encoder \
142+
--cache_latents \
143+
--instance_prompt="a photo of sks dog" \
144+
--resolution=1024 \
145+
--train_batch_size=1 \
146+
--guidance_scale=1 \
147+
--use_8bit_adam \
148+
--gradient_accumulation_steps=4 \
149+
--optimizer="adamW" \
150+
--learning_rate=1e-4 \
151+
--report_to="wandb" \
152+
--lr_scheduler="constant" \
153+
--lr_warmup_steps=100 \
154+
--max_train_steps=500 \
155+
--validation_prompt="A photo of sks dog in a bucket" \
156+
--validation_epochs=25 \
157+
--seed="0" \
158+
--push_to_hub
159+
```
160+
161+
To better track our training experiments, we're using the following flags in the command above:
162+
163+
* `report_to="wandb` will ensure the training runs are tracked on [Weights and Biases](https://wandb.ai/site). To use it, be sure to install `wandb` with `pip install wandb`. Don't forget to call `wandb login <your_api_key>` before training if you haven't done it before.
164+
* `validation_prompt` and `validation_epochs` to allow the script to do a few validation inference runs. This allows us to qualitatively check if the training is progressing as expected.
165+
166+
> [!NOTE]
167+
> If you want to train using long prompts with the T5 text encoder, you can use `--max_sequence_length` to set the token limit. The default is 77, but it can be increased to as high as 512. Note that this will use more resources and may slow down the training in some cases.
168+
169+
## LoRA + DreamBooth
170+
171+
[LoRA](https://huggingface.co/docs/peft/conceptual_guides/adapter#low-rank-adaptation-lora) is a popular parameter-efficient fine-tuning technique that allows you to achieve full-finetuning like performance but with a fraction of learnable parameters.
172+
173+
Note also that we use PEFT library as backend for LoRA training, make sure to have `peft>=0.6.0` installed in your environment.
174+
175+
### Prodigy Optimizer
176+
Prodigy is an adaptive optimizer that dynamically adjusts the learning rate learned parameters based on past gradients, allowing for more efficient convergence.
177+
By using prodigy we can "eliminate" the need for manual learning rate tuning. read more [here](https://huggingface.co/blog/sdxl_lora_advanced_script#adaptive-optimizers).
178+
179+
to use prodigy, first make sure to install the prodigyopt library: `pip install prodigyopt`, and then specify -
180+
```bash
181+
--optimizer="prodigy"
182+
```
183+
> [!TIP]
184+
> When using prodigy it's generally good practice to set- `--learning_rate=1.0`
185+
186+
To perform DreamBooth with LoRA, run:
187+
188+
```bash
189+
export MODEL_NAME="black-forest-labs/FLUX.2-dev"
190+
export INSTANCE_DIR="dog"
191+
export OUTPUT_DIR="trained-flux2-lora"
192+
193+
accelerate launch train_dreambooth_lora_flux.py \
194+
--pretrained_model_name_or_path=$MODEL_NAME \
195+
--instance_data_dir=$INSTANCE_DIR \
196+
--output_dir=$OUTPUT_DIR \
197+
--do_fp8_training \
198+
--gradient_checkpointing \
199+
--remote_text_encoder \
200+
--cache_latents \
201+
--instance_prompt="a photo of sks dog" \
202+
--resolution=512 \
203+
--train_batch_size=1 \
204+
--guidance_scale=1 \
205+
--gradient_accumulation_steps=4 \
206+
--optimizer="prodigy" \
207+
--learning_rate=1. \
208+
--report_to="wandb" \
209+
--lr_scheduler="constant_with_warmup" \
210+
--lr_warmup_steps=100 \
211+
--max_train_steps=500 \
212+
--validation_prompt="A photo of sks dog in a bucket" \
213+
--validation_epochs=25 \
214+
--seed="0" \
215+
--push_to_hub
216+
```
217+
218+
### LoRA Rank and Alpha
219+
Two key LoRA hyperparameters are LoRA rank and LoRA alpha.
220+
- `--rank`: Defines the dimension of the trainable LoRA matrices. A higher rank means more expressiveness and capacity to learn (and more parameters).
221+
- `--lora_alpha`: A scaling factor for the LoRA's output. The LoRA update is scaled by lora_alpha / lora_rank.
222+
- lora_alpha vs. rank:
223+
This ratio dictates the LoRA's effective strength:
224+
lora_alpha == rank: Scaling factor is 1. The LoRA is applied with its learned strength. (e.g., alpha=16, rank=16)
225+
lora_alpha < rank: Scaling factor < 1. Reduces the LoRA's impact. Useful for subtle changes or to prevent overpowering the base model. (e.g., alpha=8, rank=16)
226+
lora_alpha > rank: Scaling factor > 1. Amplifies the LoRA's impact. Allows a lower rank LoRA to have a stronger effect. (e.g., alpha=32, rank=16)
227+
228+
> [!TIP]
229+
> A common starting point is to set `lora_alpha` equal to `rank`.
230+
> Some also set `lora_alpha` to be twice the `rank` (e.g., lora_alpha=32 for lora_rank=16)
231+
> to give the LoRA updates more influence without increasing parameter count.
232+
> If you find your LoRA is "overcooking" or learning too aggressively, consider setting `lora_alpha` to half of `rank`
233+
> (e.g., lora_alpha=8 for rank=16). Experimentation is often key to finding the optimal balance for your use case.
234+
235+
### Target Modules
236+
When LoRA was first adapted from language models to diffusion models, it was applied to the cross-attention layers in the Unet that relate the image representations with the prompts that describe them.
237+
More recently, SOTA text-to-image diffusion models replaced the Unet with a diffusion Transformer(DiT). With this change, we may also want to explore
238+
applying LoRA training onto different types of layers and blocks. To allow more flexibility and control over the targeted modules we added `--lora_layers`- in which you can specify in a comma separated string
239+
the exact modules for LoRA training. Here are some examples of target modules you can provide:
240+
- for attention only layers: `--lora_layers="attn.to_k,attn.to_q,attn.to_v,attn.to_out.0"`
241+
- to train the same modules as in the fal trainer: `--lora_layers="attn.to_k,attn.to_q,attn.to_v,attn.to_out.0,attn.add_k_proj,attn.add_q_proj,attn.add_v_proj,attn.to_add_out,ff.net.0.proj,ff.net.2,ff_context.net.0.proj,ff_context.net.2"`
242+
- to train the same modules as in ostris ai-toolkit / replicate trainer: `--lora_blocks="attn.to_k,attn.to_q,attn.to_v,attn.to_out.0,attn.add_k_proj,attn.add_q_proj,attn.add_v_proj,attn.to_add_out,ff.net.0.proj,ff.net.2,ff_context.net.0.proj,ff_context.net.2,norm1_context.linear, norm1.linear,norm.linear,proj_mlp,proj_out"`
243+
> [!NOTE]
244+
> `--lora_layers` can also be used to specify which **blocks** to apply LoRA training to. To do so, simply add a block prefix to each layer in the comma separated string:
245+
> **single DiT blocks**: to target the ith single transformer block, add the prefix `single_transformer_blocks.i`, e.g. - `single_transformer_blocks.i.attn.to_k`
246+
> **MMDiT blocks**: to target the ith MMDiT block, add the prefix `transformer_blocks.i`, e.g. - `transformer_blocks.i.attn.to_k`
247+
> [!NOTE]
248+
> keep in mind that while training more layers can improve quality and expressiveness, it also increases the size of the output LoRA weights.
249+
250+
251+
252+
## Training Image-to-Image
253+
254+
Flux.2 lets us perform image editing as well as image generation. We provide a simple script for image-to-image(I2I) LoRA fine-tuning in [train_dreambooth_lora_flux2_img2img.py](./train_dreambooth_lora_flux2_img2img.py) for both T2I and I2I. The optimizations discussed above apply this script, too.
255+
256+
**important**
257+
258+
**Important**
259+
To make sure you can successfully run the latest version of the image-to-image example script, we highly recommend installing from source, specifically from the commit mentioned below. To do this, execute the following steps in a new virtual environment:
260+
261+
```bash
262+
git clone https://github.com/huggingface/diffusers
263+
cd diffusers
264+
pip install -e .
265+
266+
To start, you must have a dataset containing triplets:
267+
268+
* Condition image - the input image to be transformed.
269+
* Target image - the desired output image after transformation.
270+
* Instruction - a text prompt describing the transformation from the condition image to the target image.
271+
272+
[kontext-community/relighting](https://huggingface.co/datasets/kontext-community/relighting) is a good example of such a dataset. If you are using such a dataset, you can use the command below to launch training:
273+
274+
```bash
275+
accelerate launch train_dreambooth_lora_flux2_img2img.py \
276+
--pretrained_model_name_or_path=black-forest-labs/FLUX.2-dev \
277+
--output_dir="flux2-i2i" \
278+
--dataset_name="kontext-community/relighting" \
279+
--image_column="output" --cond_image_column="file_name" --caption_column="instruction" \
280+
--do_fp8_training \
281+
--gradient_checkpointing \
282+
--remote_text_encoder \
283+
--cache_latents \
284+
--resolution=1024 \
285+
--train_batch_size=1 \
286+
--guidance_scale=1 \
287+
--gradient_accumulation_steps=4 \
288+
--gradient_checkpointing \
289+
--optimizer="adamw" \
290+
--use_8bit_adam \
291+
--cache_latents \
292+
--learning_rate=1e-4 \
293+
--lr_scheduler="constant_with_warmup" \
294+
--lr_warmup_steps=200 \
295+
--max_train_steps=1000 \
296+
--rank=16\
297+
--seed="0"
298+
```
299+
300+
More generally, when performing I2I fine-tuning, we expect you to:
301+
302+
* Have a dataset `kontext-community/relighting`
303+
* Supply `image_column`, `cond_image_column`, and `caption_column` values when launching training
304+
305+
### Misc notes
306+
307+
* By default, we use `mode` as the value of `--vae_encode_mode` argument. This is because Kontext uses `mode()` of the distribution predicted by the VAE instead of sampling from it.
308+
### Aspect Ratio Bucketing
309+
we've added aspect ratio bucketing support which allows training on images with different aspect ratios without cropping them to a single square resolution. This technique helps preserve the original composition of training images and can improve training efficiency.
310+
311+
To enable aspect ratio bucketing, pass `--aspect_ratio_buckets` argument with a semicolon-separated list of height,width pairs, such as:
312+
313+
`--aspect_ratio_buckets="672,1568;688,1504;720,1456;752,1392;800,1328;832,1248;880,1184;944,1104;1024,1024;1104,944;1184,880;1248,832;1328,800;1392,752;1456,720;1504,688;1568,672"
314+
`
315+
Since Flux.2 finetuning is still an experimental phase, we encourage you to explore different settings and share your insights! 🤗

0 commit comments

Comments
 (0)