save_file in a streaming way

[wfjsw]

I'm trying to convert a 7 GB pickle file to safetensors on a Colab notebook with 12 GB memory. Initially it took ~8G memory on torch.load and got another spike when running safetensors.torch.save_file. Eventually the notebook crashed. Is it possible to avoid memory spiking by streaming it to disk?

[Narsil]

There's an open PR for that: #134

Far from finished, and we're targetting LLMs with it (so multi node/multi process) but it could fit the use case you are mentionning !