Problem with provided example validation input in the Flux Control finetuning example #12078
Description
Describe the bug
The help page for the Flux control finetuning example, https://github.com/huggingface/diffusers/blob/main/examples/flux-control/README.md, provides a sample validation input, a pose condition image
The pose conditioned model trained by the script does not process this image properly because it is in BGR format, apparent when comparing it to the openpose spec:
.
It doesn't appear that the validation image is loaded in BGR format properly, in the below line:
In my personal experiments, the validation output does not make sense. Below is an example of what my run uploaded to wandb:
Reproduction
I ran the below in the command line:
accelerate launch --config_file=/mnt/localssd/huggingface/accelerate/deepspeed.yaml train_control_lora_flux.py \
--pretrained_model_name_or_path="black-forest-labs/FLUX.1-dev" \
--dataset_name="raulc0399/open_pose_controlnet" \
--output_dir="/mnt/localssd/pose-control-lora" \
--mixed_precision="bf16" \
--train_batch_size=1 \
--rank=64 \
--gradient_accumulation_steps=4 \
--gradient_checkpointing \
--use_8bit_adam \
--learning_rate=1e-4 \
--report_to="wandb" \
--lr_scheduler="constant" \
--lr_warmup_steps=0 \
--max_train_steps=5000 \
--validation_image="openpose.png" \
--validation_prompt="A couple, 4k photo, highly detailed" \
--seed="0" \
--cache_dir="/mnt/localssd/huggingface"
Logs
System Info
- 🤗 Diffusers version: 0.34.0
- Platform: Linux-5.10.223-212.873.amzn2.x86_64-x86_64-with-glibc2.35
- Running on Google Colab?: No
- Python version: 3.10.8
- PyTorch version (GPU?): 2.7.1+cu126 (True)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Huggingface_hub version: 0.34.3
- Transformers version: 4.54.1
- Accelerate version: 1.9.0
- PEFT version: 0.17.0
- Bitsandbytes version: 0.46.1
- Safetensors version: 0.5.3
- xFormers version: not installed
- Accelerator: NVIDIA A100-SXM4-80GB, 81920 MiB
NVIDIA A100-SXM4-80GB, 81920 MiB
NVIDIA A100-SXM4-80GB, 81920 MiB
NVIDIA A100-SXM4-80GB, 81920 MiB
NVIDIA A100-SXM4-80GB, 81920 MiB
NVIDIA A100-SXM4-80GB, 81920 MiB
NVIDIA A100-SXM4-80GB, 81920 MiB
NVIDIA A100-SXM4-80GB, 81920 MiB
- Using GPU in script?: Yes.
- Using distributed or parallel set-up in script?: Yes.
Who can help?
No response