runez.decompress() seems to have bug related to file permissions against zip files

[X-Engineer-001]

This is a source base zip from https://github.com/ThomasDickey/ncurses-snapshots:
ncurses-snapshots-6_5_20250927.zip
If you use runez.decompress() in Python to decompress it, you'll find that a lot of files that should be executable become non-executable, including the important "./configure". This makes portable-python fail because it utilizes runez for decompressing.

Furthermore, this is a tarball:
ncurses-6.5.tar.gz
If you decompress it with linux command tar then compress it to a .zip with linux command zip, then decompress it again with runez.decompress() in Python, the issue also occurs.

However, decompressing the zip files with linux command unzip provides no issues.

[X-Engineer-001]

The reason that people hadn't found this issue yet is probably that "all default source URL portable-python gets external libraries from" are tarballs, no zip files.

[zsimic]

I don't compile ncurses usually, but rather let the build use the system readline as that is very common (all the OS-es I use have that). In theory portable-python should be able to compile ncurses, but you might have found an edge case indeed there.

[X-Engineer-001]

The true problem here is actually not directly related to portable-python, it's just a consequence.
The true problem is that runez.decompress(), when used to decompress zip files, will result in that the permission of files extracted being wrong.

Whether it's ncurses or not, as long as it's a zip, there's a chance that something go wrong.

[X-Engineer-001]

Opps... merely unzipping with ZipFile.extractall() from the "zipfile" std library yields the same behavior...
runez.decompress() utilizes ZipFile.extractall(), that's why...
Is there any other things we can use to unzip properly instead?
Calling the linux shell unzip is bad, because it will yields security issues...

[zsimic]

Most things are tarballs, including ncurses: https://gnu.mirror.constant.com/ncurses/
I'd rather see ZipFile.extractall() be fixed, it's not worth I think going out of our way for now to fix whatever this is (didn't notice so far that there was an issue with ZipFile.extractall())

[X-Engineer-001]

Yeah I found some explanations and workarounds like:
https://stackoverflow.com/a/39296577
This seems to be a complex issue:
python/cpython#59999

But IMO apparently it's going to be a long wait if we just sit.
Some scenario do need compilation from zip to work, and I'm currently facing one:
There's an issue about ncurses that's fixed only after 2024/12/14, that's why I can't get source codebase from https://gnu.mirror.constant.com/ncurses/ as it doesn't archive the minor tags.

What do you think about the "function overriding" workaround approach?

[zsimic]

Can you repackage that ncurses from a .zip file to a .tar.gz and republish it to one of your servers? This is good practice in any case at a company (grab the artifact and store it locally so that you have a trace as to what was used for the build)

[X-Engineer-001]

Well, portable-python only support http/https source download if I didn't get it wrong. Technically, we could create a server, store that and provide http(s) URL just for this. But it's not cost efficient.
Also this is just an example when a fine unzip function is needed. Solving my actual problem doesn't imply that this situation won't cause other troubles...
Btw it's really surprising! I didn't know even the python standard library unzip have issue all these years!

[zsimic]

Me neither, didn't know ZipFile.extractall() had an issue. I didn't get a chance to look at it closely, but will keep this issue open and see eventually to get to the bottom of it. I thought zip files would work ...