Improve `tarfile` streaming mode to handle very large archives

[za3k]

Bug report

Bug description:

I am trying to use tarfile to write very large archives, and the process is being killed by the OOM killer. I would expect+want to be able to use it in streaming mode (write an unlimited number of files) like the standard tar command line utility supports by default.

Reproduction case attached.

import gc
import io
import os
import psutil
import tarfile

if __name__ == "__main__":
    t = tarfile.open("a.tar", mode="w") # default compresslevel 9
    for i in range(1,100_000_000):
        if i % 10_000 == 0:
            gc.collect()
            process = psutil.Process(os.getpid())
            mem_info = process.memory_info()
            mem = mem_info.rss
            print(f"Iteration {i}, memory usage: {mem}")

        bs = (" "*1000 + str(i)).encode('utf8')
        with io.BytesIO(bs) as file:
            tarinfo = tarfile.TarInfo(name="cool_files/{i}.txt")
            tarinfo.size = len(bs)
            t.addfile(tarinfo, file)

The memory usage increases without bound because the list of this line in addfile():

self.members.append(tarinfo)

I'm not sure what the use-case is for this line. In write-only mode, it does not seem useful. Maybe mixed read/write? But in general it does not seem correct to assume you can fit all the tarinfo's in memory.

Edit: As a workaround, I'm setting t.members=[] manually.

CPython versions tested on:

3.13

Operating systems tested on:

Linux

[picnixz]

Well... I don't really know if it's possible to change this. Shouldn't all the members be eventually needed to avoid some duplication issues? how does tar handle this? do they have some temporary buffer that they eventually merge before exiting? or some other magic?

Also, why would you have a SINGLE tar for 100 million files? when is it a realistic use-case? (when is the memory limit hit?)

cc @jaraco

[emmatyping]

The memory usage increases without bound because the list of this line in addfile():

self.members.append(tarinfo)

I tried using streaming (mode="w|") and to skip this line if streaming but I still see significant memory growth. It seems like most of the memory being consumed is coming from creating the PAX header? Though I think that should be disposed of after the buffer to output is created. Strange...

[za3k]

I'm writing 55 million Go games in SGF format, this is a real problem I'm hitting. Trust me, as bad as it seems, having 55 million loose files is worse.

I believe tar just writes the file a second time and does a duplication (but lazily didn't check). Keep in mind, the Tape Archive format was designed around streaming. In fact, it's designed so that you can update a file in the tar by appending a second copy at the end:

Notice, that newer files don't replace their old archive copies, but instead are appended to the end of archive. The resulting archive can thus contain several members of the same name, corresponding to various versions of the same file.

Edit: I agree that most users probably won't hit out-of-memory errors, but I think it's good to reduce memory usage aside from actual crashes too.