Directly save checkpoint to specified disk location

[HarveyYan]

Description & Motivation

Can lightning save the model checkpoints directly to the target location on disk, instead of first writing it to some tmp dir like /tmp/blah then trying to move it to the target location?

Pitch

No response

Alternatives

No response

Additional context

No response

cc @lantiga @Borda

[SkafteNicki]

Hi @HarveyYan, thanks for raising this issue. Could you kindly provide a bit more information on how you are saving checkpoints? Lightning uses an atomic_save strategy which saves checkpoints to an in-memory BytesIO buffer, then writes directly to the target file using fsspec transactions, which prevents the creating of incomplete checkpoint. There is nothing in the process that should save to a tmp dir.