zipfile detection of when to write a zip64 header should be made accurate

[gpshead]

Bug report

Proposal:

Today our zipfile module internal implementation uses a heuristic dance to determine when a zip64 header is likely to be required between zipfile.ZipFile._open_to_write() and zipfile._ZipWriteFile.close().

This seems rather silly. Any the time zipfile._ZipWriteFile.close() is called, we know the real uncompressed and compressed data sizes and can deterministically decide at that time. Instead of the existing heuristic of "if the expected input file_size * 1.05 > ZIP64_LIMIT" used within _open_to_write() today.

The only time we should ever raise an exception regarding zip64 being requires is if the API user has explicitly forbidden zip64's use.

I wouldn't backport this change to a stable release as it will alter the exact output produced in some circumstances (zip64 headers will no longer be added unnecessarily in borderline cases where they were not needed), but it is fair to consider it more of a bug that removes an odd API internal implementation wart as well as a feature.

Has this already been discussed elsewhere?

This is a minor feature, which does not need previous discussion elsewhere

Links to previous discussion of this feature:

No response

[gpshead]

Amending: This is not always true as implemented. _open_to_write needs to add the initial inline zip file header as the header size depends on zip64 or not before a _ZipWriteFile is created to fill in the data which merely seeks back and updates the same header to fill in the CRC and sizes.

To get out of the heuristic business we either need to:
A. always write a zip64 header
B. handle the rare boundary condition when the compressed data winds up larger than uncompressed specially by rewriting things in that situation
C. give up on the zip format shenanigans, keep our heuristic, and go shopping.

[gpshead]

investigating which approaches other zip creation tools use would be informative rather than reinventing the wheels here.

[serhiy-storchaka]

Always writing a zip64 header is inefficient for small files.

[thatch]

The start of compressed data is different when we have to write a zip64 extra, which is what serhiy is pointing at above.

@gpshead

Since there's no PR I'm not sure what your exact approach was here, but if you're suggesting buffer the compressed data before writing the LFH that's basically what main...thatch:cpython:unseekable_writestr does. I'm concerned that while it works great for small files (or zips whose output is already kept in memory) this would increase the memory use drastically for most normal use, and enhancing to detect when the boundary is crossed and flushing seems like a hard-to-test change.

My vote is to go shopping for now, the heuristic is pretty good and we now support zip64 in zipimport which was the only reason I personally ever set allowZip64=False.