Skip to content

Commit e8a6b01

Browse files
strickvlhtahir1
strickvl
and
htahir1
authored
Feature/paris ai tinkerers (#130)
* add code start * updates * updates * format * formatting * update * add notebook * add README * Added paris project * Add frontend app with styling and user authentication * Add instructions for running the app * use modal step operator * updates * formatting * updates * updates * ignore big folders * updates * more updates * add k8s version * fix dockersettings * updated order * update gitignore * upload latest images * renaming * more renaming * use flux * formatting * add flux script * updates and formatting * working version * refactor * update with gif gen step * bugfixes * return bytes * fix dimensions * maybe final for today? * update visualization + add imports * add some images to gitignore * update avoiding OOM error * working pipeline * remove class reference * remove class reference * update for hamza * add hamza images * pull out username * switch cache off * bump training steps * add more images * reduce the steps * update training code * actually FT dev doh! * chore: Update training code and reduce the number of steps * feat: Refactor variable name in k8s_run.py The variable `prompt` in the `train_model` function has been refactored to `instance_prompt` for clarity and consistency. This change improves the readability of the code and makes it easier to understand the purpose of the variable. * update training params * update training params * update training steps * update scripts * update modal code * Add enable_cache=False setting to train_model function * update modal run and requirements * Update learning rate to 0.0004 and remove triton from dependencies * add more photos * disable cache * disable cache again * resize images * remove dog * update * another * Hamza configs * Refactored * Refactored * ignore faces data locally * Refactored * Refactored * formatting and add data copy part * Deleted data * Update max_train_steps to 1000 * ;latest * ;latest * update modal * use develop * update * use hamza id * formatting * add multi-video script * update pipeline * final before run * updated prompts * Cleaned up * Cleaned up * Cleaned up * Cleaned up * Cleaned up * Cleaned up * revert visualisation changes * Cleaned up * update README * add some links * add missing asset --------- Co-authored-by: Hamza Tahir <[email protected]> Co-authored-by: Hamza Tahir <[email protected]>
1 parent c25cf81 commit e8a6b01

25 files changed

+8287
-0
lines changed

‎flux-dreambooth/.gitignore‎

Lines changed: 174 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,174 @@
1+
# Byte-compiled / optimized / DLL files
2+
__pycache__/
3+
*.py[cod]
4+
*$py.class
5+
6+
# C extensions
7+
*.so
8+
9+
# Distribution / packaging
10+
.Python
11+
build/
12+
develop-eggs/
13+
dist/
14+
downloads/
15+
eggs/
16+
.eggs/
17+
lib/
18+
lib64/
19+
parts/
20+
sdist/
21+
var/
22+
wheels/
23+
share/python-wheels/
24+
*.egg-info/
25+
.installed.cfg
26+
*.egg
27+
MANIFEST
28+
29+
# PyInstaller
30+
# Usually these files are written by a python script from a template
31+
# before PyInstaller builds the exe, so as to inject date/other infos into it.
32+
*.manifest
33+
*.spec
34+
35+
# Installer logs
36+
pip-log.txt
37+
pip-delete-this-directory.txt
38+
39+
# Unit test / coverage reports
40+
htmlcov/
41+
.tox/
42+
.nox/
43+
.coverage
44+
.coverage.*
45+
.cache
46+
nosetests.xml
47+
coverage.xml
48+
*.cover
49+
*.py,cover
50+
.hypothesis/
51+
.pytest_cache/
52+
cover/
53+
54+
# Translations
55+
*.mo
56+
*.pot
57+
58+
# Django stuff:
59+
*.log
60+
local_settings.py
61+
db.sqlite3
62+
db.sqlite3-journal
63+
64+
# Flask stuff:
65+
instance/
66+
.webassets-cache
67+
68+
# Scrapy stuff:
69+
.scrapy
70+
71+
# Sphinx documentation
72+
docs/_build/
73+
74+
# PyBuilder
75+
.pybuilder/
76+
target/
77+
78+
# Jupyter Notebook
79+
.ipynb_checkpoints
80+
81+
# IPython
82+
profile_default/
83+
ipython_config.py
84+
85+
# pyenv
86+
# For a library or package, you might want to ignore these files since the code is
87+
# intended to run in multiple environments; otherwise, check them in:
88+
# .python-version
89+
90+
# pipenv
91+
# According to pypa/pipenv#598, it is recommended to include Pipfile.lock in version control.
92+
# However, in case of collaboration, if having platform-specific dependencies or dependencies
93+
# having no cross-platform support, pipenv may install dependencies that don't work, or not
94+
# install all needed dependencies.
95+
#Pipfile.lock
96+
97+
# poetry
98+
# Similar to Pipfile.lock, it is generally recommended to include poetry.lock in version control.
99+
# This is especially recommended for binary packages to ensure reproducibility, and is more
100+
# commonly ignored for libraries.
101+
# https://python-poetry.org/docs/basic-usage/#commit-your-poetrylock-file-to-version-control
102+
#poetry.lock
103+
104+
# pdm
105+
# Similar to Pipfile.lock, it is generally recommended to include pdm.lock in version control.
106+
#pdm.lock
107+
# pdm stores project-wide configurations in .pdm.toml, but it is recommended to not include it
108+
# in version control.
109+
# https://pdm.fming.dev/latest/usage/project/#working-with-version-control
110+
.pdm.toml
111+
.pdm-python
112+
.pdm-build/
113+
114+
# PEP 582; used by e.g. github.com/David-OConnor/pyflow and github.com/pdm-project/pdm
115+
__pypackages__/
116+
117+
# Celery stuff
118+
celerybeat-schedule
119+
celerybeat.pid
120+
121+
# SageMath parsed files
122+
*.sage.py
123+
124+
# Environments
125+
.env
126+
.venv
127+
env/
128+
venv/
129+
ENV/
130+
env.bak/
131+
venv.bak/
132+
133+
# Spyder project settings
134+
.spyderproject
135+
.spyproject
136+
137+
# Rope project settings
138+
.ropeproject
139+
140+
# mkdocs documentation
141+
/site
142+
143+
# mypy
144+
.mypy_cache/
145+
.dmypy.json
146+
dmypy.json
147+
148+
# Pyre type checker
149+
.pyre/
150+
151+
# pytype static type analyzer
152+
.pytype/
153+
154+
# Cython debug symbols
155+
cython_debug/
156+
157+
# PyCharm
158+
# JetBrains specific template is maintained in a separate JetBrains.gitignore that can
159+
# be found at https://github.com/github/gitignore/blob/main/Global/JetBrains.gitignore
160+
# and can be added to the global gitignore or merged into this file. For a more nuclear
161+
# option (not recommended) you can uncomment the following to ignore the entire idea folder.
162+
#.idea/
163+
164+
165+
# Specific to this project
166+
models/
167+
data/dog-class/
168+
diffusers/
169+
.venv/
170+
sd-dreambooth-blupus/
171+
data/cat-class/
172+
data/dog/
173+
data/aria/
174+
data/hamza-faces/

‎flux-dreambooth/README.md‎

Lines changed: 72 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,72 @@
1+
# Flux DreamBooth: Personalized AI with ZenML
2+
3+
This project was motivated by a desire to explore the limits of the latest AI
4+
technologies, particularly the Flux models and the Stable Diffusion img2vid
5+
model. By finetuning the Flux.1-dev model on images of the ZenML co-founder
6+
using DreamBooth, we aimed to showcase the potential for personalized AI
7+
applications.
8+
9+
![](assets/batch-dreambooth.png)
10+
11+
[DreamBooth](https://huggingface.co/docs/diffusers/v0.30.2/training/dreambooth) is a technique that allows for the creation of custom text-to-image models by finetuning a pre-trained model on a small dataset of images featuring a specific subject. In this case, we used a dataset of cropped portrait photos of the ZenML co-founder to create a personalized model capable of generating novel images of him in various styles and contexts.
12+
13+
To make the finetuning process more efficient, we employed LoRA (Low-Rank Adaptation) adapters. LoRA allows for faster finetuning by only updating a small fraction of the model's weights, which can then be saved separately from the original model. The resulting LoRA adapters for this project have been pushed to the Hugging Face repository at [https://huggingface.co/strickvl/flux-dreambooth-hamza](https://huggingface.co/strickvl/flux-dreambooth-hamza).
14+
15+
While we finetuned the [Flux.1-dev](https://huggingface.co/black-forest-labs/FLUX.1-dev) model for training, we switched to the [Flux-Schnell](https://huggingface.co/spaces/black-forest-labs/FLUX.1-schnell) model for inference. Flux-Schnell is a faster variant of the Flux model that maintains comparable quality, making it an excellent choice for efficient batch inference. The LoRA adapters worked seamlessly with Flux-Schnell, allowing us to generate high-quality personalized images quickly.
16+
17+
Crafting effective prompts was a crucial aspect of this project. We experimented
18+
with various prompts to find those that worked best with the finetuned model.
19+
Additionally, we had to tailor the prompts to the limitations of our dataset,
20+
which consisted primarily of cropped portrait photos. By carefully designing
21+
prompts that played to the strengths of the available data, we were able to
22+
generate impressive personalized images and short animated videos.
23+
24+
<video width="512" height="512" controls autoplay loop>
25+
<source src="assets/hamza_superman.mp4" type="video/mp4">
26+
Your browser does not support the video tag.
27+
</video>
28+
29+
30+
The pipeline outputs a grouped image showcasing the results of various prompts
31+
and generates short 3-second animated videos using the
32+
[stable-video-diffusion-img2vid-xt](https://huggingface.co/stabilityai/stable-video-diffusion-img2vid-xt)
33+
img2vid model. These outputs demonstrate the potential for creating engaging,
34+
personalized content using state-of-the-art AI technologies.
35+
36+
## Getting Started
37+
38+
To get started with this project, follow these steps:
39+
40+
1. Clone the ZenML Projects repository: `git clone https://github.com/zenml-io/zenml-projects.git`
41+
2. Navigate to the `flux-dreambooth` directory: `cd zenml-projects/flux-dreambooth`
42+
3. Install the required dependencies: `pip install -r requirements.txt`
43+
4. For a guided walkthrough of the code, check out the `walkthrough.ipynb` notebook.
44+
45+
## Stack Setup
46+
47+
You'll need a cloud stack to run these pipelines, and you'll need GPU quotas so
48+
as to be able to run the training and batch inference steps. You can get started
49+
with a 1-click deployment on all the major cloud providers using [our simple
50+
guide](https://docs.zenml.io/how-to/stack-deployment/deploy-a-cloud-stack).
51+
52+
## Running the pipeline
53+
54+
To run the pipeline, use one of the following commands depending on your stack:
55+
- For Kubernetes: `python k8s_run.py`
56+
- For Modal: `python modal_run.py`
57+
58+
## Project Structure
59+
60+
The project contains the following files with comments:
61+
62+
- `configs/k8s_run_refactored_multi_video.yaml`: Configuration files used in the notebooks
63+
- `k8s_run.py`: Script to run the pipeline on a Kubernetes stack
64+
- `modal_run.py`: Script to run the pipeline on a Modal stack
65+
- `test_examples_utils.py`: Related to the diffusers script used in older versions of the pipeline
66+
- `train_dreambooth_lora_flux.py`: From the diffusers library, used in older versions of the pipeline
67+
- `train_dreambooth.py`: From the diffusers library, used in older versions of the pipeline
68+
- `walkthrough.ipynb`: Jupyter Notebook providing a guided walkthrough of the code
69+
70+
## Contributing
71+
72+
Contributions to this project are welcome! If you find any issues or have suggestions for improvements, please open an issue or submit a pull request.

‎flux-dreambooth/assets/app_screenshot_1.png‎

855 KB
Loading

‎flux-dreambooth/assets/app_screenshot_2.png‎

64 KB
Loading

‎flux-dreambooth/assets/app_screenshot_3.png‎

363 KB
Loading

‎flux-dreambooth/assets/batch-dreambooth.png‎

1.03 MB
Loading

‎flux-dreambooth/assets/blupus-demo.png‎

593 KB
Loading

‎flux-dreambooth/assets/hamza_superman.mp4‎

211 KB
Binary file not shown.

‎flux-dreambooth/configs/k8s_run_refactored_multi_video.yaml‎

Lines changed: 143 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,143 @@
1+
# Specify the "ZenML Model"
2+
model:
3+
name: "flux_personalized_model"
4+
audience: "ML Engineers and Researchers"
5+
description: "Personalized image generation model using DreamBooth and Flux.1"
6+
ethics: "Ensure responsible use of generated images"
7+
license: "Apache 2.0"
8+
limitations: "Limited to the trained subject and style"
9+
tags: ["dreambooth", "flux", "stable-diffusion-image2video"]
10+
11+
# Parameters of the pipeline
12+
parameters:
13+
instance_example_dir: "az://demo-zenmlartifactstore/hamza-faces"
14+
instance_name: "sks htahir1"
15+
class_name: "man"
16+
model_name: "black-forest-labs/FLUX.1-dev"
17+
hf_username: "htahir1"
18+
hf_repo_suffix: "flux-dreambooth-hamza"
19+
prefix: "A portrait photo of"
20+
resolution: 512
21+
train_batch_size: 1
22+
rank: 32
23+
gradient_accumulation_steps: 1
24+
learning_rate: 0.0002
25+
lr_scheduler: "constant"
26+
lr_warmup_steps: 0
27+
max_train_steps: 1300
28+
push_to_hub: True
29+
checkpointing_steps: 1000
30+
seed: 117
31+
32+
# Per step configuration
33+
steps:
34+
train_model:
35+
enable_cache: True
36+
# Real-time settings for Docker and resources
37+
settings:
38+
# Controls Docker building
39+
docker:
40+
parent_image: "pytorch/pytorch:2.2.2-cuda11.8-cudnn8-runtime"
41+
environment:
42+
PJRT_DEVICE: "CUDA"
43+
USE_TORCH_XLA: "false"
44+
MKL_SERVICE_FORCE_INTEL: 1
45+
HF_TOKEN: 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
46+
HF_HOME: "/mnt/data"
47+
python_package_installer: "uv"
48+
requirements: "requirements.txt"
49+
python_package_installer_args:
50+
system: null
51+
apt_packages: ["git", "ffmpeg", "gifsicle"]
52+
53+
# Kubernetes settings
54+
orchestrator.kubernetes:
55+
pod_settings:
56+
affinity:
57+
nodeAffinity:
58+
requiredDuringSchedulingIgnoredDuringExecution:
59+
nodeSelectorTerms:
60+
- matchExpressions:
61+
- key: "zenml.io/gpu"
62+
operator: "In"
63+
values: ["yes"]
64+
volumes:
65+
- name: "data-volume"
66+
persistentVolumeClaim:
67+
claimName: "pvc-managed-premium"
68+
volume_mounts:
69+
- name: "data-volume"
70+
mountPath: "/mnt/data"
71+
72+
batch_inference:
73+
settings:
74+
# Controls Docker building
75+
docker:
76+
parent_image: "pytorch/pytorch:2.2.2-cuda11.8-cudnn8-runtime"
77+
environment:
78+
PJRT_DEVICE: "CUDA"
79+
USE_TORCH_XLA: "false"
80+
MKL_SERVICE_FORCE_INTEL: 1
81+
HF_TOKEN: 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
82+
HF_HOME: "/mnt/data"
83+
python_package_installer: "uv"
84+
requirements: "requirements.txt"
85+
python_package_installer_args:
86+
system: null
87+
apt_packages: ["git", "ffmpeg", "gifsicle"]
88+
89+
# Kubernetes settings
90+
orchestrator.kubernetes:
91+
pod_settings:
92+
affinity:
93+
nodeAffinity:
94+
requiredDuringSchedulingIgnoredDuringExecution:
95+
nodeSelectorTerms:
96+
- matchExpressions:
97+
- key: "zenml.io/gpu"
98+
operator: "In"
99+
values: ["yes"]
100+
volumes:
101+
- name: "data-volume"
102+
persistentVolumeClaim:
103+
claimName: "pvc-managed-premium"
104+
volume_mounts:
105+
- name: "data-volume"
106+
mountPath: "/mnt/data"
107+
108+
image_to_video:
109+
enable_cache: False
110+
settings:
111+
# Controls Docker building
112+
docker:
113+
parent_image: "pytorch/pytorch:2.2.2-cuda11.8-cudnn8-runtime"
114+
environment:
115+
PJRT_DEVICE: "CUDA"
116+
USE_TORCH_XLA: "false"
117+
MKL_SERVICE_FORCE_INTEL: 1
118+
HF_TOKEN: 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
119+
HF_HOME: "/mnt/data"
120+
python_package_installer: "uv"
121+
requirements: "requirements.txt"
122+
python_package_installer_args:
123+
system: null
124+
apt_packages: ["git", "ffmpeg", "gifsicle"]
125+
126+
# Kubernetes settings
127+
orchestrator.kubernetes:
128+
pod_settings:
129+
affinity:
130+
nodeAffinity:
131+
requiredDuringSchedulingIgnoredDuringExecution:
132+
nodeSelectorTerms:
133+
- matchExpressions:
134+
- key: "zenml.io/gpu"
135+
operator: "In"
136+
values: ["yes"]
137+
volumes:
138+
- name: "data-volume"
139+
persistentVolumeClaim:
140+
claimName: "pvc-managed-premium"
141+
volume_mounts:
142+
- name: "data-volume"
143+
mountPath: "/mnt/data"

‎flux-dreambooth/constants.py‎

Whitespace-only changes.

0 commit comments

Comments
 (0)