Weird face generation with latest fast-DreamBooth colab

[juan9999]

I’ve previously trained 6+ faces and had no issues using a variant of 1.5.

This time I'm trying to train 14 512x512 pictures of a face(mostly shoulder and close up faces, (MODEL FCE) grouped into 3 categories (FCE1 (1-5), FCE2(6-10), FCE3(11-14)

I’ve tried following the recommendations “These default settings are for a dataset of 10 pictures which is enough for training a face, start with 650 or lower, test the model, if not enough, resume training for 150 steps, keep testing until you get the desired output,” - Since I was training 14, I tested with 750, 1000, and 1500 and got similar results.

Less than 10% of generated images whereas the rest have melted faces.

Has anything changed in fast-Dreambooth or am I doing something stupid?

Also, when I tried resuming training from 1000, it seems to have not used any of the previous saved checkpoints

Resuming Training...
Training the UNet...
'########:'########:::::'###::::'####:'##::: ##:'####:'##::: ##::'######:::
... ##..:: ##.... ##:::'## ##:::. ##:: ###:: ##:. ##:: ###:: ##:'##... ##::
::: ##:::: ##:::: ##::'##:. ##::: ##:: ####: ##:: ##:: ####: ##: ##:::..:::
::: ##:::: ########::'##:::. ##:: ##:: ## ## ##:: ##:: ## ## ##: ##::'####:
::: ##:::: ##.. ##::: #########:: ##:: ##. ####:: ##:: ##. ####: ##::: ##::
::: ##:::: ##::. ##:: ##.... ##:: ##:: ##:. ###:: ##:: ##:. ###: ##::: ##::
::: ##:::: ##:::. ##: ##:::: ##:'####: ##::. ##:'####: ##::. ##:. ######:::
:::..:::::..:::::..::..:::::..::....::..::::..::....::..::::..:::......::::

Progress:|███████ | 28% 557/2000 [08:44<22:24, 1.07it/s, loss=0.0723, lr=1.45e-5]

[TheLastBen]

2000 steps is too much, reduce the unet learning rate to 5e-6

[juan9999]

Thanks I had tested all the checkpoints from 750 to 2000 (what used to work before).

I will try back down to 650 with increments of 150 with 5e-6

[juan9999]

Ok, so I retested with 5e-6 and 650 steps and it was too little training steps (with 14 pictures) so I increased to 1000 and it seems to have done the trick! Thanks for the suggestions.

A suggestion that may help me and others is to potentially include a version number at the beginning of the colab file with a link to the relevant read.me or announcement of changes in the workbook.