Checkpoint saves successfully then immediately OOMs at end of training primary save

[mashdragon]

Is there any reason model saving could OOM (on CUDA) doing the "primary save" at the end of training, immediately after it successfully saved a checkpoint?

[NanoCode012]

Do you have the log for this? Could it just be a coincidence (aka can you replicate it) ? Does it occur even if you move the save point a few steps after?

[NanoCode012]

@mashdragon ,

How can I configure axolotl to save its logs?

You could log to tensorboard. That one should be local AFAIK.

use_tensorboard: true

or

...  --use-tensorboard

Strangely I had saves_per_epoch commented out:

I noticed it's also a float. Currently, we only accept int for this config.

It just seems weird that the "second save" so to speak would consume too much extra GPU memory.

Does this mean, the checkpoint training passed successfully, it was the save post-training that OOM-ed?

And by moving the save point what do you mean (maybe show what config settings you want)

That's okay, I just wanted to verify if it was during training or right after.

[mashdragon]

I noticed it's also a float. Currently, we only accept int for this config.

Oh yes, that's probably why.

Does this mean, the checkpoint training passed successfully, it was the save post-training that OOM-ed?

Yes, so there was a checkpoint that saved successfully at the end of training, and then the save post-training happened immediately after (no training steps in between to my knowledge) and this post-training save OOM-ed.

[NanoCode012]

I see. In the meantime, using that final checkpoint should work as it's the last step. If you do experience this again, please let us know. This time, it could just be random.

[mashdragon]

Thanks - I think I ended up just using the final checkpoint as you suggested.

[NanoCode012]

Thanks for getting back. Will close this discussion, but if something new comes up, please leave a message.