Best Training configurations for Faces

[cloneofsimo]

My take on the learing rate,
really not anything conclusive, but seems like even higher-learning rate can work as well. (upto 4e-4)

prompt = "female 3d game character bnha, Skill magic geek inside matrix deepdream radiating a glowing aura stuff loot legends stylized digital illustration video game icon artstation lois van baarle, ilya kuvshinov, rossdraws", $\alpha = 1.0$

prompt = "portrait of female 3d game character bnha, impressionist style from the 19th century, claude monet, oil painting", $\alpha = 0.6$

All with num_inference_steps=50, guidance_scale=4.5, Euler A scheduler

Columns represents 1000 steps,
Rows represent learning rates, ["2.5e-5", "5e-5", "1e-4", "2e-4", "4e-4"] starting from top.

[brian6091]

Interesting, are you running a constant learning rate schedule without warmup?

[sitatec]

@cian0 were you able to find a config that work well for human faces with concepts like cyberpunk or steampunk?

[cian0]

@sitatec honestly the results are varying, sometimes 8k steps work with one face, then wouldn't even capture another, so I'm assuming that lora might work mostly on faces that are already similar to what's already inside the model. I'm reverting to shiv's for now

[brian6091]

@cian0 , would you happen to have a small dataset for a face where LoRA isn't doing well? I had some difficulty with a person that I made sure wasn't in the SD model, and I wonder if it's my source data.

[cian0]

@brian6091 sure, though i couldn't share them publicly, you're in Dreambooth discord as well right? I'll add you there and send some test dataset when i get back to my workstation

[brian6091]

@cian0 yeah, on the Dreambooth discord. Thanks!

[cloneofsimo]

Pivotal Tuning

Rank1, fine-tunes FF as well.
Every image represents 500 steps, inversion happens until 2500 steps, fine-tuning kicks in after.

2e-6

5e-6

1e-5

2e-5

[cian0]

So 6k @ 2e-5 seems the best?

[cloneofsimo]

More results on Inversion:
So technically not related to LoRA, but...

lr : 3e-4

1r : 1e-3

[brian6091]

Going off the rails at 1e-3!

[ExponentialML]

Looks like a lower learning rate benefits from using pivotal tuning.