State of investigations re tox install issues

[aecay]

I'm writing this issue to document the efforts I've made to unpick the reason(s) why tox is not working correctly on jenkins.

The first issue is that the python version included in the jenkins slave image (which is not upgradable from alpine upstream as of this writing) vendors an old version of pip in the new virtualenvs it creates. pypa/virtualenv#2214. This version of pip has at least two problems:

• it has a crappy dependency resolver
• it doesn't understand binary packages for alpine

This means that it will get stuck on ujson which it can't find a compatible binary package of (and which it's also not willing to build from source for mysterious reasons) and will spend ~forever trying various combinations of packages to get it to work, eventually failing. The fix for this is to explicitly install a newer, fixed version of pip by adding VIRTUALENV_PIP=21.3.1 to the passenv block in tox.ini.

However, when this happens a second problem rears its head. The binary distribution of ujson for alpine (musl libc) on pypi has the included shared library misnamed:

$ unzip ujson-4.2.0-cp39-cp39-musllinux_1_1_x86_64.whl 
Archive:  ujson-4.2.0-cp39-cp39-musllinux_1_1_x86_64.whl
   creating: ujson.libs/
   creating: ujson-4.2.0.dist-info/
  inflating: ujson.cpython-39-x86_64-linux-gnu.so  ### <<<<<
  inflating: ujson.libs/libgcc_s-a04fdf82.so.1  
  inflating: ujson.libs/libstdc++-a9383cce.so.6.0.28  
  inflating: ujson-4.2.0.dist-info/WHEEL  
  inflating: ujson-4.2.0.dist-info/top_level.txt  
  inflating: ujson-4.2.0.dist-info/LICENSE.txt  
  inflating: ujson-4.2.0.dist-info/RECORD  
  inflating: ujson-4.2.0.dist-info/METADATA  

It should be ...-linux-musl.so. Reported upstream: ultrajson/ultrajson#485

I've run out of time to debug this tonight, so I'm recording what I've been doing here so I can come back to it.