Skip to content

Commit 15f5002

Browse files
acoracleacoracle
committed
Addes flux-1-finetuning
1 parent 73bad83 commit 15f5002

File tree

6 files changed

+616
-0
lines changed

6 files changed

+616
-0
lines changed

‎cloud-infrastructure/ai-infra-gpu/ai-infrastructure/flux-1-finetuning/LICENSE

Lines changed: 35 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,35 @@
1+
Copyright (c) 2021, 2023 Oracle and/or its affiliates.
2+
3+
The Universal Permissive License (UPL), Version 1.0
4+
5+
Subject to the condition set forth below, permission is hereby granted to any
6+
person obtaining a copy of this software, associated documentation and/or data
7+
(collectively the "Software"), free of charge and under any and all copyright
8+
rights in the Software, and any and all patent rights owned or freely
9+
licensable by each licensor hereunder covering either (i) the unmodified
10+
Software as contributed to or provided by such licensor, or (ii) the Larger
11+
Works (as defined below), to deal in both
12+
13+
(a) the Software, and
14+
(b) any piece of software and/or hardware listed in the lrgrwrks.txt file if
15+
one is included with the Software (each a "Larger Work" to which the Software
16+
is contributed by such licensors),
17+
18+
without restriction, including without limitation the rights to copy, create
19+
derivative works of, display, perform, and distribute the Software and make,
20+
use, sell, offer for sale, import, export, have made, and have sold the
21+
Software and the Larger Work(s), and to sublicense the foregoing rights on
22+
either these or other terms.
23+
24+
This license is subject to the following condition:
25+
The above copyright notice and either this complete permission notice or at
26+
a minimum a reference to the UPL must be included in all copies or
27+
substantial portions of the Software.
28+
29+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
30+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
31+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
32+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
33+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
34+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
35+
SOFTWARE.

‎cloud-infrastructure/ai-infra-gpu/ai-infrastructure/flux-1-finetuning/README.md

Lines changed: 139 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,139 @@
1+
# flux-1-finetuning
2+
3+
This repo is designed to quickly prepare a demo that showcases how to use OCI GPU shapes to finetune lora models of flux.1-dev.
4+
5+
Flux is a set of diffusion models created by BlackForest Labs, and are subject to [licensing terms](https://github.com/black-forest-labs/flux/blob/main/model_licenses/LICENSE-FLUX1-dev)
6+
In this blog we will show how to fine tune flux.1-dev that is available in [HuggingFace](https://huggingface.co/black-forest-labs/FLUX.1-dev)
7+
8+
There are several projects that can be used to work with flux models
9+
- [ComfyUI](https://github.com/comfyanonymous/ComfyUI) is a powerful and user-friendly tool for creating high-quality images using AI, including the FLUX model. It offers a modular workflow design that allows users to create custom image generation processes by connecting different components
10+
- [AItoolkit](https://github.com/comfyanonymous/ComfyUI) is a tool that simplifies Flux fine tuning experience expoecially to reduce VRAM requirements.
11+
- [SimpleTuner](https://github.com/bghira/SimpleTuner) is a set of scripts that simplify distributed fine tuning on multiple GPUs
12+
13+
Prerequisites:
14+
- Linux based GPU VM with recent Nvidia driver and Cuda toolkit
15+
- git, Miniconda installed
16+
- An account in HuggingFace where you can login with huggingface-cli login
17+
18+
## Installing AI toolkit ##
19+
20+
use aitoolkit.yaml to prepare a conda environment with the required packages
21+
22+
```
23+
conda create env -f aitoolkit.yaml
24+
conda activate aitoolkit
25+
```
26+
27+
then you can clone the ai-toolkit
28+
```
29+
git clone https://github.com/ostris/ai-toolkit.git
30+
cd ai-toolkit
31+
git submodule update --init --recursive
32+
```
33+
34+
## Dataset generation ##
35+
36+
WIP
37+
you can take 10 pictures of yourself .
38+
39+
40+
## Training
41+
42+
Aitoolkit has a large set of options that can be exploited to train a lora model for flux1. You can fin examples in the directory config/examples/.
43+
According to the different GPUs you can use them to eitther reduce video memory consumption, or to improve training performance.
44+
45+
46+
- folder_path: "/path/to/images/folder" , speicfy where the dataset is
47+
- gradient_checkpointing: true This feature allows to reduce memory footprint, but this increases computation time by about 35%. On large memory GPUs it is convenient to set it to false.
48+
- model:quantize: true This uses intermediate 8bit quantization to reduce memory footprint, the final model will still be 16bits, so turn it on only on small GPUs.
49+
- model:low_vram: true This further reduces memory footprint on very small GPUs.
50+
- prompts: this is a list of prompts that are used the create intermidiate images to chck quality, for analyzing performances you can remove them
51+
- batch_size: 1 increasing batch size on a single GPU deteriorates performance, recommended to stick with 1.
52+
- trigger_word: a GPU Specialist Here you set the keyword that you can use in the prompt.
53+
54+
## Installing ComfyUI
55+
56+
ComfyUI can be used the test the generated lora model. It can be installed in the same conda env as Attoolkit
57+
58+
```
59+
git clone https://github.com/comfyanonymous/ComfyUI/tree/v0.3.10
60+
cd ComfyUI
61+
python main.py
62+
```
63+
64+
You can connect to the ComfyUI GUI by pointing your browser to port 8188, according to the network configuration a port forward might be required.
65+
66+
Then you need to import models that are required by the workflow.
67+
68+
Download the [Clip Safetensor](https://huggingface.co/comfyanonymous/flux_text_encoders/blob/main/clip_l.safetensors) to ComfyUI/models/clip/
69+
This model plays a crucial role in text-to-image generation tasks by processing and encoding textual input.
70+
71+
Download the Text Encoder [T5xxl Safetensor](https://huggingface.co/comfyanonymous/flux_text_encoders/blob/main/t5xxl_fp8_e4m3fn.safetensors) to ComfyUI/models/clip/
72+
73+
Download the [VAE Safetensor](https://huggingface.co/black-forest-labs/FLUX.1-schnell/blob/main/ae.safetensors) to ComfyUI/models/vae
74+
75+
Download the [Flux.1-dev UNET model](https://huggingface.co/black-forest-labs/FLUX.1-dev/tree/main) to ComfyUI/models/unet
76+
77+
## Testing Lora models with ComfyUI
78+
79+
Everytime you create a lora model with Ai-toolkit you can copy it to ComfyUI/models/lora
80+
81+
Import the workflow by opening the file workflow-lora.json
82+
83+
You will then be able to select the model in the Load Lora box. Make sure also the proper models are selected in the Load diffusion Model, DualCLIPLoader, and Load VAE boxes.
84+
85+
You can write your own prompt in the CLIP Text Encode box, remeber to refer to the keyword used for training the Lora.
86+
![Alt text](files/ComfyUI.png?raw=true "ComfyUI Lora workflow")
87+
88+
## Installing SimpleTuner
89+
90+
```
91+
git clone --branch=release https://github.com/bghira/SimpleTuner.git
92+
```
93+
Copy config/config.json.example to config/config.json
94+
95+
Then you execute the training with
96+
97+
```
98+
./train.sh
99+
```
100+
101+
Parallel training is possible using Accelerate (the Deepspeed implementation on Flux is buggy at the time of writing.
102+
When more GPUs are used, the batch size is increased automatically, so the number os steps required to process one full epoch is riduced proportionally.
103+
104+
105+
If present the Accelerate configuration will be taken from the config file in
106+
107+
~/.cache/huggingface/accelerate/default_config.yaml
108+
109+
```
110+
compute_environment: LOCAL_MACHINE
111+
debug: false
112+
distributed_type: *MULTI_GPU*
113+
downcast_bf16: 'no'
114+
enable_cpu_affinity: true
115+
gpu_ids: all
116+
machine_rank: 0
117+
main_training_function: main
118+
mixed_precision: bf16
119+
num_machines: 1
120+
num_processes: 4
121+
rdzv_backend: static
122+
same_network: true
123+
tpu_env: []
124+
tpu_use_cluster: false
125+
tpu_use_sudo: false
126+
use_cpu: false
127+
```
128+
129+
If this file is not present you can create a file config/config.env and use it to set this environmental variable:
130+
131+
```
132+
TRAINING_NUM_PROCESSES=4
133+
```
134+
135+
136+
137+
138+
139+

‎cloud-infrastructure/ai-infra-gpu/ai-infrastructure/flux-1-finetuning/files/ComfyUI.png

143 KB
Loading

‎cloud-infrastructure/ai-infra-gpu/ai-infrastructure/flux-1-finetuning/files/aitoolkit.yaml

Lines changed: 173 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,173 @@
1+
name: aitoolkit
2+
channels:
3+
- defaults
4+
- https://repo.anaconda.com/pkgs/main
5+
- https://repo.anaconda.com/pkgs/r
6+
dependencies:
7+
- _libgcc_mutex=0.1=main
8+
- _openmp_mutex=5.1=1_gnu
9+
- bzip2=1.0.8=h5eee18b_6
10+
- ca-certificates=2024.11.26=h06a4308_0
11+
- ld_impl_linux-64=2.40=h12ee557_0
12+
- libffi=3.4.4=h6a678d5_1
13+
- libgcc-ng=11.2.0=h1234567_1
14+
- libgomp=11.2.0=h1234567_1
15+
- libstdcxx-ng=11.2.0=h1234567_1
16+
- libuuid=1.41.5=h5eee18b_0
17+
- ncurses=6.4=h6a678d5_0
18+
- openssl=3.0.15=h5eee18b_0
19+
- pip=24.2=py311h06a4308_0
20+
- python=3.11.10=he870216_0
21+
- readline=8.2=h5eee18b_0
22+
- setuptools=75.1.0=py311h06a4308_0
23+
- sqlite=3.45.3=h5eee18b_0
24+
- tk=8.6.14=h39e8969_0
25+
- wheel=0.44.0=py311h06a4308_0
26+
- xz=5.4.6=h5eee18b_1
27+
- zlib=1.2.13=h5eee18b_1
28+
- pip:
29+
- absl-py==2.1.0
30+
- accelerate==1.2.1
31+
- aiofiles==23.2.1
32+
- albucore==0.0.16
33+
- albumentations==1.4.15
34+
- annotated-types==0.7.0
35+
- antlr4-python3-runtime==4.9.3
36+
- anyio==4.7.0
37+
- attrs==24.3.0
38+
- bitsandbytes==0.45.0
39+
- certifi==2024.12.14
40+
- charset-normalizer==3.4.0
41+
- clean-fid==0.1.35
42+
- click==8.1.7
43+
- clip-anytorch==2.6.0
44+
- controlnet-aux==0.0.7
45+
- dctorch==0.1.2
46+
- diffusers==0.32.0.dev0
47+
- docker-pycreds==0.4.0
48+
- einops==0.8.0
49+
- eval-type-backport==0.2.0
50+
- fastapi==0.115.6
51+
- ffmpy==0.5.0
52+
- filelock==3.16.1
53+
- flatten-json==0.1.14
54+
- fsspec==2024.12.0
55+
- ftfy==6.3.1
56+
- gitdb==4.0.11
57+
- gitpython==3.1.43
58+
- gradio==5.9.1
59+
- gradio-client==1.5.2
60+
- grpcio==1.68.1
61+
- h11==0.14.0
62+
- hf-transfer==0.1.8
63+
- httpcore==1.0.7
64+
- httpx==0.28.1
65+
- huggingface-hub==0.27.0
66+
- idna==3.10
67+
- imageio==2.36.1
68+
- importlib-metadata==8.5.0
69+
- invisible-watermark==0.2.0
70+
- jinja2==3.1.4
71+
- jsonmerge==1.9.2
72+
- jsonschema==4.23.0
73+
- jsonschema-specifications==2024.10.1
74+
- k-diffusion==0.1.1.post1
75+
- kornia==0.7.4
76+
- kornia-rs==0.1.7
77+
- lazy-loader==0.4
78+
- lpips==0.1.4
79+
- lycoris-lora==1.8.3
80+
- markdown==3.7
81+
- markdown-it-py==3.0.0
82+
- markupsafe==2.1.5
83+
- mdurl==0.1.2
84+
- mpmath==1.3.0
85+
- networkx==3.4.2
86+
- ninja==1.11.1.3
87+
- numpy==1.26.4
88+
- nvidia-cublas-cu12==12.4.5.8
89+
- nvidia-cuda-cupti-cu12==12.4.127
90+
- nvidia-cuda-nvrtc-cu12==12.4.127
91+
- nvidia-cuda-runtime-cu12==12.4.127
92+
- nvidia-cudnn-cu12==9.1.0.70
93+
- nvidia-cufft-cu12==11.2.1.3
94+
- nvidia-curand-cu12==10.3.5.147
95+
- nvidia-cusolver-cu12==11.6.1.9
96+
- nvidia-cusparse-cu12==12.3.1.170
97+
- nvidia-nccl-cu12==2.21.5
98+
- nvidia-nvjitlink-cu12==12.4.127
99+
- nvidia-nvtx-cu12==12.4.127
100+
- omegaconf==2.3.0
101+
- open-clip-torch==2.29.0
102+
- opencv-python==4.10.0.84
103+
- opencv-python-headless==4.10.0.84
104+
- optimum-quanto==0.2.4
105+
- orjson==3.10.12
106+
- oyaml==1.0
107+
- packaging==24.2
108+
- pandas==2.2.3
109+
- peft==0.14.0
110+
- pillow==11.0.0
111+
- platformdirs==4.3.6
112+
- prodigyopt==1.1.1
113+
- protobuf==5.29.2
114+
- psutil==6.1.1
115+
- pydantic==2.10.4
116+
- pydantic-core==2.27.2
117+
- pydub==0.25.1
118+
- pygments==2.18.0
119+
- python-dateutil==2.9.0.post0
120+
- python-dotenv==1.0.1
121+
- python-multipart==0.0.20
122+
- python-slugify==8.0.4
123+
- pytorch-fid==0.3.0
124+
- pytz==2024.2
125+
- pywavelets==1.8.0
126+
- pyyaml==6.0.2
127+
- referencing==0.35.1
128+
- regex==2024.11.6
129+
- requests==2.32.3
130+
- rich==13.9.4
131+
- rpds-py==0.22.3
132+
- ruff==0.8.4
133+
- safehttpx==0.1.6
134+
- safetensors==0.4.5
135+
- scikit-image==0.25.0
136+
- scipy==1.14.1
137+
- semantic-version==2.10.0
138+
- sentencepiece==0.2.0
139+
- sentry-sdk==2.19.2
140+
- setproctitle==1.3.4
141+
- shellingham==1.5.4
142+
- six==1.17.0
143+
- smmap==5.0.1
144+
- sniffio==1.3.1
145+
- starlette==0.41.3
146+
- sympy==1.13.1
147+
- tensorboard==2.18.0
148+
- tensorboard-data-server==0.7.2
149+
- text-unidecode==1.3
150+
- tifffile==2024.12.12
151+
- timm==1.0.12
152+
- tokenizers==0.21.0
153+
- toml==0.10.2
154+
- tomlkit==0.13.2
155+
- torch==2.5.1
156+
- torchdiffeq==0.2.5
157+
- torchsde==0.2.6
158+
- torchvision==0.20.1
159+
- tqdm==4.67.1
160+
- trampoline==0.1.2
161+
- transformers==4.47.1
162+
- triton==3.1.0
163+
- typer==0.15.1
164+
- typing-extensions==4.12.2
165+
- tzdata==2024.2
166+
- urllib3==2.2.3
167+
- uvicorn==0.34.0
168+
- wandb==0.19.1
169+
- wcwidth==0.2.13
170+
- websockets==14.1
171+
- werkzeug==3.1.3
172+
- zipp==3.21.0
173+
prefix: /home/ubuntu/anaconda3/envs/aitoolkit2

0 commit comments

Comments
 (0)