gh-126834: Properly read zip64 archives with non-empty zip64 extensible data sector in Zip64 end of central directory record

[VladRassokhin]

Fixes #126834

• Issue: Cannot open zip64 file if Zip64EOCD record has additional data #126834

[bedevere-app]

Most changes to Python require a NEWS entry. Add one using the blurb_it web app or the blurb command-line tool.

If this change has little impact on Python users, wait for a maintainer to apply the skip news label instead.

[cmaloney]

I think struct.unpack returns relative to the struct, and that will be at offset - sizeEndCentDir64Locator (where it was read from). SEEK_SET / 0 isn't right as reloff is a relative position from where the struct was read, not an absolute position in the file.

[VladRassokhin]

Although the variable is named reloff and the spec states "relative offset of the zip64 end of central directory record" without specifying relative to what, in reality it's offset from the beginning of the file. See code which writes it

cpython/Lib/zipfile/__init__.py

Line 2078 in 2313f84

stringEndArchive64Locator, 0, pos2, 1)

Also, the same in the libzip code: https://github.com/nih-at/libzip/blob/d0ebf7fa268ae2e59e575cb3a72e6bc259e3fdd8/lib/zip_open.c#L853

[VladRassokhin]

@cmaloney wdyt on renaming reloff, worth changing to e.g. eocd_offset?

[cmaloney]

Definitely would be clearer to me, but not sure it's worth the extra noise in the diff though / additional lines changed.

[danifus]

The offsets are complicated by data that may precede the start of the zip content which is why some of the tests are failing. reloff is the number of bytes from the start of the first local file header in the zip which may not be the actual start of the file. I can't think of a good way of directly computing the start of the zip64 end of record block if there's data preceding the start of the zip part of the file. Might have to do a bit of a search.

    # Assume no 'zip64 extensible data'
    fpin.seek(offset - sizeEndCentDir64Locator - sizeEndCentDir64, 2)
    data = fpin.read(sizeEndCentDir64)
    if len(data) != sizeEndCentDir64:
        return endrec

    sig, sz, create_version, read_version, disk_num, disk_dir, \
        dircount, dircount2, dirsize, diroffset = \
        struct.unpack(structEndArchive64, data)
    if sig != stringEndArchive64:
        loc_pos = fpin.tell()
        # Seek to the earliest possible eocd64 start
        fpin.seek(reloff, 0)
        # Read all the data between here and the eocd64 locator
        data = fpin.read(loc_pos - reloff)
        start = data.rfind(stringEndArchive64)
        if start >= 0 and len(data) - start > sizeEndCentDir64:
            sig, sz, create_version, read_version, disk_num, disk_dir, \
                dircount, dircount2, dirsize, diroffset = \
                struct.unpack(structEndArchive64, data[start:start+sizeEndCentDir64])
            if sig != stringEndArchive64:
                return endrec
        else:
            return endrec

My code needs improvement as data = fpin.read(loc_pos - reloff) might read a substantial amount of data if there's a big blob of data before the zip. It would also be a good idea to check that the sizes of the offsets are consistent with regards to:

• the zip64 end locator is actually sz bytes from the position just after the sz field
• The position of the stringEndArchive64 signature found using rfind is the same position as the stringEndArchive64 signature that is found directly after the central directory.

[bedevere-app]

Most changes to Python require a NEWS entry. Add one using the blurb_it web app or the blurb command-line tool.

If this change has little impact on Python users, wait for a maintainer to apply the skip news label instead.

[cmaloney]

This will need a news entry describing the feature it's adding, the CI / PR testing failures look related to the code changes and need to get resolved

[bedevere-app]

Most changes to Python require a NEWS entry. Add one using the blurb_it web app or the blurb command-line tool.

If this change has little impact on Python users, wait for a maintainer to apply the skip news label instead.

[python-cla-bot]

The following commit authors need to sign the Contributor License Agreement:

• [email protected]