ckpt: rank_zero_only

[Borda]

Before submitting

• Was this discussed/agreed via a Github issue? (no need for typos and docs improvements)
• Did you read the contributor guideline, Pull Request section?
• Did you make sure to update the docs?
• Did you write any new necessary tests?

What does this PR do?

Fixes uploading the same model multiple times
Will add integration tests when aded multi-GPU CI

PR review

Anyone in the community is free to review the PR once the tests have passed.
If we didn't discuss your PR in GitHub issues there's a high chance it will not be merged.

Did you have fun?

Make sure you had fun coding 🙃

[justusschock]

Would be good to have a test for this. Mock the API and run with distributed on CPU.

[Borda]

Would be good to have a test for this. Mock the API and run with distributed on CPU.

started to work on adding such test, but mocking the ddp_spawn processes is not trivial, so I will merge this one as it as after validation on GPU and continue work on adding a CPU-distributed test