How do I get resume training to start at last checkpoint or complete model?

[juan9999]

I'm trying to resume a dreambooth training to see if I had undertrained a model.

It had last stopped at 650, there was a checkpoint saved at 500 and wanted to continue the training but it seems to start from zero. What am I doing wrong?

[TheLastBen]

load the session and choose the saved checkpoint, any training after that would be a continuation to the previous training

[juan9999]

Thanks, I'm assuming your recommendations are if I had closed my colab session.

Within the same colab session, I had:

Trained the mode
Then ran Test The Trained Model to test it

I wasn't happy with the results so I decided to train it some more, so I stopped the execution of Test The Trained Model

I then went to the Start DreamBooth step, checked resume training, increased the UNet_Training_Steps to 1000 (from 650), and set Text_Encoder_Concept_Training_Steps to zero and restarted that step.

But it did not reuse the last .ckpt file or any of the checkpoints.

[TheLastBen]

if it writes "Resuming Training" in green, that means it's using the previous model.

[juan9999]

ok so just to confirm resume training doesn’t resume training at the last step it restarts training?

or am i misunderstanding and it is resuming (which preciously stopped at step 650) and going for 1000 more steps?

it would be great to have a txt file produced that lists how the metadata model was trained kinda like the metadata for generated images which includes the prompts.

[TheLastBen]

Yes even if it starts from 0, as long as it displays "resuming" that means it continues training.

Implementing a metadata file is on the todo list.

[SorenTruelsen]

ok so just to confirm resume training doesn’t resume training at the last step it restarts training?

or am i misunderstanding and it is resuming (which preciously stopped at step 650) and going for 1000 more steps?

it would be great to have a txt file produced that lists how the metadata model was trained kinda like the metadata for generated images which includes the prompts.

It's a bit confusing, lets say I train a model 1000 times. Then I resume, the counter will start at 0 and overwrite the already existing checkpoints (200, 400, 600, 800) with the same name but the new model. So, the one called 200 is actually 1200 etc.

[TheLastBen]

Yes, if you want to keep the previous models, just move them manually in your gdrive