Sharing our configs for suggestions and ideas

[ddpasa]

I wanted to start a thread where we share our training configs. This can maybe help newcomers start more quickly, and also enable us to get feedback on our configs. I'll start by sharing my most recent training.

This is a rather large (rank=128) flux lora that I'm training on 48 GB card using a dataset of size approximately 20k images (split approximately evenly between train and reg datasets).

seed = 1234567

output_name = "lora_#####"                  # Trained model output name
train_data_dir = #####
output_dir = #####
reg_data_dir = #####
logging_dir = #####
sample_prompts = #####
save_model_as = "safetensors"
save_precision = "bf16"                    # Save model precision (fp16, bf16, float)
pretrained_model_name_or_path = "#####/flux1-dev-fp8.safetensors"
fp8_base = true
ae = "#####/ae.safetensors"
clip_l = "#####/clip_l.safetensors"
t5xxl = "#####/t5xxl_fp16.safetensors"

huggingface_repo_type = "model"
huggingface_repo_id = ############
huggingface_path_in_repo = ############
huggingface_token = ############
async_upload = true
huggingface_repo_visibility = "private"

optimizer_args = [ "scale_parameter=False", "relative_step=False", "warmup_init=False", ]
optimizer_type = "Adafactor"
train_batch_size = 8
gradient_accumulation_steps = 1
unet_lr = 2.5e-4
text_encoder_lr = [ 7.5e-5, 7.5e-5, ]
fused_backward_pass = false
max_grad_norm = 0

enable_bucket = true
bucket_no_upscale = true
bucket_reso_steps = 64
max_bucket_reso = 2048
min_bucket_reso = 256
cache_latents = true
cache_latents_to_disk = true
caption_extension = ".txt"
enable_wildcard = true
#caption_dropout_rate = 0.025

network_alpha = 64
network_dim = 128

scale_weight_norms = 50

timestep_sampling = "shift"
discrete_flow_shift = 3.1582

blocks_to_swap = 0

epoch = 10
max_train_epochs = 10

save_every_n_epochs = 1
sample_every_n_steps = 200
save_every_n_steps = 600
sample_sampler = "euler"


apply_t5_attn_mask = false
dynamo_backend = "no"
full_bf16 = true
gradient_checkpointing = true
guidance_scale = 1.0
highvram = true
huber_c = 0.1
huber_schedule = "exponential"
loss_type = "l2"
max_timestep = 1000
mem_eff_save = true
mixed_precision = "bf16"
model_prediction_type = "raw"
network_args = [ "train_t5xxl=True", "train_clip_l=True" ]
network_module = "networks.lora_flux"
prior_loss_weight = 1
resolution = "1024,1024"
persistent_data_loader_workers = true
max_data_loader_n_workers = 1
vae_batch_size = 16
xformers = true

This works okayish. The foregrounds are good, but I get quality decay in the backgrounds. Some observations are:

• fp16 training outputs NaNs. Not sure why, but bf16 is okay.
• The memory usage is around 45 GB
• I'm eager to move away from Adafactor to fancier optimizers like Prodigy or ScheduleFree, but I didn't find good parameters yet.
• fused_backward_pass was actually slower, so I turned it off. Not sure if I'm doing something wrong.
• batch size 8 is determined by my available memory (48GB). I did not run ablation studies to find the optimal values.
• I read scary things online about the t5 attention mask, so I turned it off. Not sure if it is better to leave it on.
• I played with l1 and l2 loss, but didn't find that one is obviously superiour to the other.

I'm eager to get your critique and suggestions!