CI: build Docker failing

[daniellepintz]

I noticed two CI jobs are consistently failing - CI build Docker / build-Conda (3.9, 1.9) and (3.9, 1.10)
with the error

Error: buildx failed with: error: failed to solve: executor failed running [/bin/bash -c pip install --no-cache-dir --global-option="--cuda_ext" https://github.com/NVIDIA/apex/archive/refs/heads/master.zip &&     python -c "from apex import amp"]: exit code: 1

https://github.com/PyTorchLightning/pytorch-lightning/runs/3483024689
https://github.com/PyTorchLightning/pytorch-lightning/runs/3483024738

Starting this discussion to track and try to resolve. anyone have any ideas? cc @Borda

[Borda]

seems like failing to install Apex, either on the build/install or in testing whether it is correctly installed...
could you try to build it locally and check some fixing?
https://github.com/PyTorchLightning/pytorch-lightning/blob/master/dockers/README.md
as other Conda dockers are passing I would guess that the Apex is not compatible with PT 1.9+

[carmocca]

I'm seeing lots of recent issues about installation in https://github.com/NVIDIA/apex/issues

They might've just broken master.

[Borda]

Let's freeze on a version that still works...

[daniellepintz]

Thanks! I am having trouble building it locally with the command docker image build -t pytorch-lightning:base-cuda-py3.8-pt1.9 -f dockers/base-cuda/Dockerfile --build-arg PYTHON_VERSION=3.8 --build-arg PYTORCH_VERSION=1.9 . same thing with 3.7/1.6

CMake Error at /usr/share/cmake-3.10/Modules/FindPackageHandleStandardArgs.cmake:137 (message):
Could NOT find Pytorch: Found unsuitable version "", but required is at least "1.2.0" (found )

full error: https://gist.github.com/daniellepintz/4591231de0e72f9530463bd4badae742

is the command I'm running correct?

[daniellepintz]

cc @carmocca @SeanNaren who also contributed to the Docker readme https://github.com/PyTorchLightning/pytorch-lightning/blob/master/dockers/README.md