mmap can do copy on write on all major platforms

[sswam]

In your readme under Notes -> Zero-copy, it says:

Also In PyTorch/numpy, you need a mutable buffer, and we don't really want to mutate a mmaped file, so 1 copy is really necessary to use the thing freely in user code

POSIX mmap accepts a flag MAP_PRIVATE to create a private copy-on-write mapping. This works on Linux, MacOS, BSD, etc. Windows has an equivalent function CreateFileMapping which also supports copy on write.

Numpy has a function numpy.lib.format.open_memmap which abstracts over these two implementations. Here's an article describing how to use copy-on-write mmap from numpy: https://pythonspeed.com/articles/reduce-memory-array-copies/

I don't know how to do this in pytorch, but surely it's possible.

[Narsil]

Hi @sswam.

I'll try and modify this text to be a bit more accurate.

This comment is actually slightly outdated. torch does use copy-on-write: https://pytorch.org/docs/stable/storage.html#torch.TypedStorage.from_file. But it doesn't take advantage of that in its pickled format.

Thanks for the link on open_mmap. Afaik it works only on individual arrays, not zipped dict or array, correct ? (Individual arrays is just too cumbersome to work with in ML in practice, so only npz is actually in question).

That being said, you can use mmapped buffers (so zero-copy) to create those tensors in both libraries.