Failing CI for python3.6 MacOS

[observingClouds]

The plain conda installation of python 3.6 installs since a few days certifi as an additional dependency, in particular conda-forge/osx-64::certifi-2016.9.26-py36_0 on a Mac. This dependency causes the CI currently to fail because since pip 10 packages need to be actively be deinstalled before being upgraded. (see e.g. this comment). Some package(s) in requirements_test.txt and/or requirements.txt however need certifi>=2017.4.17 which results in the following error:

  Attempting uninstall: certifi
    Found existing installation: certifi 2016.9.26
ERROR: Cannot uninstall 'certifi'. It is a distutils installed project and thus we cannot accurately determine which files belong to it which would lead to only a partial uninstall.

Minimal, reproducible code sample, a copy-pastable example if possible

The error can be reproduced locally on a Mac by running a simplified version of CI job defined in

numcodecs/.github/workflows/ci-osx.yaml

Lines 37 to 40 in 15f1713

	conda create -n env python=${{matrix.python-version}} wheel pip compilers 'clang>=12.0.1'
	conda activate env
	which pip
	pip install -r requirements_test.txt -r requirements.txt

conda create -n env python=3.6
conda activate env
pip install -r requirements_test.txt -r requirements.txt

The log of the failed CI job is here.

Possible solution

To fix this issue, one possible solution that I came across and that seem to work, is to tell pip to ignore already installed packages by adding the flag --ignore-installed.

conda create -n env python=3.6
conda activate env
pip install -r requirements_test.txt -r requirements.txt --ignore-installed

Another possible solution might be to limit the pip version because

conda create -n env python=3.6 pip=20.0.2  # this is the current default

installs certifi, while

conda create -n env python=3.6 "pip>20.0.2"

does not.

These are my suggestions to have still python 3.6 support and a running CI that would allow merging #299.

This issue is probably not only affecting #299, but also all new/current merge requests. I expect that the CI will now fail for those as well when re-run.

[joshmoore]

Hey @observingClouds. Thanks for the heads up and all the sleuthing! I'm about to head offline for the evening, but I'll add a little context from my side, namely that I likely should have addressed support in this repo along with zarr-developers/zarr-python#765 but thoughts from everyone welcome.

[observingClouds]

Great, I just wanted to show options in case python 3.6 should still be supported. I have no strong feelings for python 3.6.

[jakirkham]

Yeah agreed. It should be dropped. This came up elsewhere. If you have time to send a PR @observingClouds, can review 🙂

[jakirkham]

cc @martindurant (since I think you had asked about this elsewhere)

[martindurant]

Happy to drop. In my case I think it might have been 3.7 already.

[jakirkham]

We've dropped Python 3.6 with PR ( #318 )