Any way to prevent disconnections or resume training for the old method?

[SAC020]

Hi,

I am on Google Colab Pro+ subscription and using the old training method.

With 130 training images at 768px, it takes around 35 hours to train a model to 100 steps, using the standard GPU (and I cannot / don't want to spend the rate for A100).

Problem is after aprox 24h I get disconnected overnight.

• (preferably) is there any way to prevent the disconnections?
• (fallback solution) is there any way to resume training on the old method, if a training session got disconnected?

Thx

[TheLastBen]

hi,
The 24h is a limit set by google, I don't think you can override that.

To resume training in case of a disconnection, set it to save intermediary checkpoints, an use the "download model" cell to load the latest model.

[SAC020]

I have set it to save intermediary checkpoints, but they are 2GB (btw, any way to save 4GB intermediary checkpoints?). Do these work for resume?

I am starting to think of running sessions of 10x or 20x steps, then using the method you have suggested to continue training.

[SAC020]

Indeed, I wasn't able to prevent Google from disconnecting me after 23hours+something

However, thinking of it, maybe it's an overall better approach training and re-training the model in small increments instead of a "big bang"?

Meaning, for example:

• in the first iteration, the class images will be based on standard SD model
• however, on the second one, the class images will be generated by the first "epoch", so I'm guessing it will be training based on an earlier version of itself; intuitively, it should yield better results after iterating on itself, than comparing a large number of steps with standard SD? Just a hunch, no idea

Is there a parameter I can specify to generate class images at a higher resolution than 512? I am training at 768, so it would make sense to me to generate the class images at the same resolution

[SlightlyMe]

Hi, I'm adding my question to this as I believe they are connected.
I also am having disconnection issues, and have been saving intermediary check points. However what I have done is rename the old model (old-model.ckpt) and renamed the latest checkpoint model.ckpt and just resumed as normal. I'm kinda guessing from your previous reply that that also would work, but would be nice to hear so.
I occasionally get good runs with colab and clock yup a lot of check points, is it safe to delete them? and if so, why aren't they automatically deleted?