Few questions / bug report about Textual Inversion

[petalas]

If it crashes / gets interrupted for any reason it seems to overwrite existing embedding and image logs, anyone else had that before?
How does it determine where to continue from?
I assumed it checked the generated filenames in the textual_inversion log folder since it seems to work fine when I want to train for extra steps after a successful run.

Exact scenario for clarity:
I have embeddings and images generated up to 56500, it crashed, I put max steps 60000 when I started it again (assuming it would generate 3500) but it started generating 1/19999.

Should I just grab the last checkpoint from the logs and replace the file in the actual embeddings folder? Are the steps included inside the file? Don't want to fuck it up as it's pretty slow to train.

[Heathen]

Should I just grab the last checkpoint from the logs and replace the file in the actual embeddings folder? Are the steps included inside the file? Don't want to fuck it up as it's pretty slow to train.

Yes and yes. Make a backup just in case.

[petalas]

Should I just grab the last checkpoint from the logs and replace the file in the actual embeddings folder? Are the steps included inside the file? Don't want to fuck it up as it's pretty slow to train.

Yes and yes. Make a backup just in case.

I see, thanks, that worked.

Suggestion for improvement: check if the last checkpoint has a higher step number than the current embedding and maybe prompt to restore etc.