Failing to load present METADATA

[mboisson]

I am writing a script using wheelfile, and it fails on some wheels that seem otherwise correct. It fails with :

Traceback (most recent call last):
  File "../manipulate_wheels.py", line 115, in <module>
    main()
  File "../manipulate_wheels.py", line 57, in main
    with WheelFile(w) as wf:
  File "/cvmfs/soft.computecanada.ca/custom/python/envs/manipulate_wheels/lib/python3.8/site-packages/wheelfile.py", line 1224, in __init__
    self.validate()
  File "/cvmfs/soft.computecanada.ca/custom/python/envs/manipulate_wheels/lib/python3.8/site-packages/wheelfile.py", line 1730, in validate
    raise ValueError(
ValueError: METADATA file is not present in the archive or is corrupted.

yet, if I unzip the wheel, there is:

jupyterlab-3.1.7.dist-info/METADATA

I am not sure what causes this.

[mboisson]

Ah, the issue is that wheelfile does not handle the License-File attribute.

That is in specific. In general, I think wheelfile barfs whenever it encounters an attribute it does not know about. Is this something expected ?

[mboisson]

Ah, I see that this is using
https://www.python.org/dev/peps/pep-0639/
which is still in draft.... but I find it is being used in many of the wheels we host already.

[MrMino]

Yea, I've just debugged it and was wondering when did PyPA specify the License-File field. The sad thing is that the wheel in question still uses Metadata-Version: 2.1...

Thanks for pointing this out. I'll probably add it provisionally into the next version even though it's not in the spec yet. Do you happen to have any other examples wheels that use this, just so I can make sure there aren't any other surprises?

[mboisson]

I will run our script
https://github.com/ComputeCanada/wheels_builder/blob/main/manipulate_wheels.py
on a large sample of our ~6k wheels, see if anything else pops up.

I have patched our local installation of wheelfile.py to include the patch in PR #14, so it won't crash for License-File. I'll see if it crashes on something else.

[MrMino]

From the discussion it seems that the current text is not up to date. I'll have to either wait for the PEP to get updated or read through the messages to get the feel for the consesnus, both of which will take time.

Thanks for the PR. Next version will come after I'll get back from vacation (at the beginning of September).

[mboisson]

Ok, I added some exception handling to wheelfile.py to be able to print a description of the error, and here is what I got:

$ ./manipulate_wheels.py --wheels /cvmfs/soft.computecanada.ca/custom/python/wheelhouse/gentoo/generic/*{cp37,py3,any}*.whl --print_req | grep -B1 Exception
Requirements for wheel /cvmfs/soft.computecanada.ca/custom/python/wheelhouse/gentoo/generic/blist-1.3.6+computecanada-cp37-cp37m-linux_x86_64.whl:
Exception:Unknown field: 'provide'.
Requirements for wheel /cvmfs/soft.computecanada.ca/custom/python/wheelhouse/gentoo/generic/Bottleneck-1.3.2+computecanada-cp37-cp37m-linux_x86_64.whl:
Exception:Unknown field: 'require'.
--
Requirements for wheel /cvmfs/soft.computecanada.ca/custom/python/wheelhouse/gentoo/generic/dipy-1.3.0+computecanada-cp37-cp37m-linux_x86_64.whl:
Exception:Unknown field: 'require'.
Requirements for wheel /cvmfs/soft.computecanada.ca/custom/python/wheelhouse/gentoo/generic/dipy-1.4.0+computecanada-cp37-cp37m-linux_x86_64.whl:
Exception:Unknown field: 'require'.
Requirements for wheel /cvmfs/soft.computecanada.ca/custom/python/wheelhouse/gentoo/generic/dipy-1.4.1+computecanada-cp37-cp37m-linux_x86_64.whl:
Exception:Unknown field: 'require'.
--
Requirements for wheel /cvmfs/soft.computecanada.ca/custom/python/wheelhouse/gentoo/generic/gvar-11.9.2+computecanada-cp37-cp37m-linux_x86_64.whl:
Exception:Unknown field: 'require'.
--
Requirements for wheel /cvmfs/soft.computecanada.ca/custom/python/wheelhouse/gentoo/generic/immutables-0.15+computecanada-cp37-cp37m-linux_x86_64.whl:
Exception:Unknown field: 'provide'.
--
Requirements for wheel /cvmfs/soft.computecanada.ca/custom/python/wheelhouse/gentoo/generic/jaxlib-0.1.64+computecanada-cp37-none-linux_x86_64.whl:
Exception:Invalid version: '0.1.64+computecanada+cuda110'
--
Requirements for wheel /cvmfs/soft.computecanada.ca/custom/python/wheelhouse/gentoo/generic/louvain-0.7.0+computecanada-cp37-cp37m-linux_x86_64.whl:
Exception:Unknown field: 'provide'.
--
Requirements for wheel /cvmfs/soft.computecanada.ca/custom/python/wheelhouse/gentoo/generic/MDAnalysis-1.0.0+computecanada-cp37-cp37m-linux_x86_64.whl:
Exception:Unknown field: 'require'.
Requirements for wheel /cvmfs/soft.computecanada.ca/custom/python/wheelhouse/gentoo/generic/MDAnalysis-1.1.1+computecanada-cp37-cp37m-linux_x86_64.whl:
Exception:Unknown field: 'provide'.
--
Requirements for wheel /cvmfs/soft.computecanada.ca/custom/python/wheelhouse/gentoo/generic/phono3py-2.0.0+computecanada-cp37-cp37m-linux_x86_64.whl:
Exception:Unknown field: 'provide'.
Requirements for wheel /cvmfs/soft.computecanada.ca/custom/python/wheelhouse/gentoo/generic/phonopy-2.11.0+computecanada-cp37-cp37m-linux_x86_64.whl:
Exception:Unknown field: 'provide'.
Requirements for wheel /cvmfs/soft.computecanada.ca/custom/python/wheelhouse/gentoo/generic/phonopy-2.9.1+computecanada-cp37-cp37m-linux_x86_64.whl:
Exception:Unknown field: 'provide'.
--
Requirements for wheel /cvmfs/soft.computecanada.ca/custom/python/wheelhouse/gentoo/generic/pysam-0.15.4+computecanada-cp37-cp37m-linux_x86_64.whl:
Exception:Unknown field: 'require'.
Requirements for wheel /cvmfs/soft.computecanada.ca/custom/python/wheelhouse/gentoo/generic/pysam-0.16.0.1+computecanada-cp37-cp37m-linux_x86_64.whl:
Exception:Unknown field: 'require'.
--
Requirements for wheel /cvmfs/soft.computecanada.ca/custom/python/wheelhouse/gentoo/generic/scikit_image-0.18.2+computecanada-cp37-cp37m-linux_x86_64.whl:
Exception:Unknown field: 'require'.
--
Requirements for wheel /cvmfs/soft.computecanada.ca/custom/python/wheelhouse/gentoo/generic/spglib-1.16.1+computecanada-cp37-cp37m-linux_x86_64.whl:
Exception:Unknown field: 'provide'.
--
Requirements for wheel /cvmfs/soft.computecanada.ca/custom/python/wheelhouse/gentoo/generic/vegas-4.0.1+computecanada-cp37-cp37m-linux_x86_64.whl:
Exception:Unknown field: 'require'.
Requirements for wheel /cvmfs/soft.computecanada.ca/custom/python/wheelhouse/gentoo/generic/vegas-4.0.3+computecanada-cp37-cp37m-linux_x86_64.whl:
Exception:Unknown field: 'require'.
--
Requirements for wheel /cvmfs/soft.computecanada.ca/custom/python/wheelhouse/gentoo/generic/ete3-3.1.2+computecanada-py3-none-any.whl:
Exception:Unknown field: 'require'.
--
Requirements for wheel /cvmfs/soft.computecanada.ca/custom/python/wheelhouse/gentoo/generic/requests_ntlm-1.1.0+computecanada-py3-none-any.whl:
Exception:Unknown field: 'provide'.
--
Requirements for wheel /cvmfs/soft.computecanada.ca/custom/python/wheelhouse/gentoo/generic/ete3-3.1.2+computecanada-py3-none-any.whl:
Exception:Unknown field: 'require'.
--
Requirements for wheel /cvmfs/soft.computecanada.ca/custom/python/wheelhouse/gentoo/generic/requests_ntlm-1.1.0+computecanada-py3-none-any.whl:
Exception:Unknown field: 'provide'.

It looks like Require and Provide are somewhat commonly used.

The invalid version one is obviously our mess up when first patching our wheels without wheelfile, using our crude sed in a shell script.

[MrMino]

Alright, thanks! I'll work it out in the next version.

[ccoulombe]

How is this coming along?

[MrMino]

Not swiftly unfortunately, due to my private life. This is still on my mind though, don't worry.