Merge branch 'ubuntu20.04'

Author: lukasmasuch

Dockerfile

@@ -572,7 +572,6 @@ Port tunneling is quite useful when you have started any server-based tool withi
 - `8090`: Jupyter server.
 - `8054`: VS Code server.
 - `5901`: VNC server.
-- `3389`: RDP server.
 - `22`: SSH server.
 
 You can find port information on all the tools in the [supervisor configuration](https://github.com/ml-tooling/ml-workspace/blob/main/resources/supervisor/supervisord.conf).
@@ -1069,6 +1068,11 @@ import sys
 You can do this, but please be aware that this port is <b>not</b> protected by the workspace's authentication mechanism then! For security reasons, we therefore highly recommend to use the <a href="#access-ports">Access Ports</a> functionality of the workspace.
 </details>
 
+<details>
+<summary><b>System and Tool Translations</b> (click to expand...)</summary>
+If you want to configure another language than English in your workspace and some tools are not translated properly, have a look <a href="https://github.com/ml-tooling/ml-workspace/issues/70#issuecomment-841863145">at this issue</a>. Try to comment out the 'exclude translations' line in `/etc/dpkg/dpkg.cfg.d/excludes` and re-install / configure the package.
+</details>
+
 ---
 
 <br>

README.md

@@ -13,7 +13,7 @@
 parser = argparse.ArgumentParser(add_help=False)
 parser.add_argument(
     "--" + FLAG_FLAVOR,
-    help="Flavor (full, light, minimal, r, spark, gpu) used for docker container",
+    help="Flavor (full, light, minimal, gpu) used for docker container",
     default="all",
 )
 
@@ -40,18 +40,9 @@
     args[FLAG_FLAVOR] = "full"
     build_utils.build(".", args)
 
-    args[FLAG_FLAVOR] = "r"
-    build_utils.build("r-flavor", args)
-
-    args[FLAG_FLAVOR] = "spark"
-    build_utils.build("spark-flavor", args)
-
     args[FLAG_FLAVOR] = "gpu"
     build_utils.build("gpu-flavor", args)
 
-    args[FLAG_FLAVOR] = "gpu-r"
-    build_utils.build("r-flavor", args)
-
     build_utils.exit_process(0)
 
 # unknown flavor -> try to build from subdirectory

build.py

@@ -1,6 +1,6 @@
 # Workspace Update Process
 
-We plan to do a full workspace image update (all libraries and tools) about every three month. The full update involves quiet a bit of manual work as documented below:
+We plan to do a full workspace image update (all libraries and tools) about every three months. The full update involves quiet a bit of manual work as documented below:
 
 1. Refactor incubation zone:
 
@@ -17,7 +17,7 @@ We plan to do a full workspace image update (all libraries and tools) about ever
 
 3. Update core (gui) tools:
 
-   - TigetVNC: [latest release](https://dl.bintray.com/tigervnc/stable/)
+   - TigerVNC: [latest release](https://dl.bintray.com/tigervnc/stable/)
    - noVNC: [latest release](https://github.com/novnc/noVNC/releases/latest)
    - Websockify: [latest release](https://github.com/novnc/websockify/releases/latest)
    - VS Code Server: [latest release](https://github.com/cdr/code-server/releases/latest)
@@ -47,18 +47,16 @@ We plan to do a full workspace image update (all libraries and tools) about ever
    - pycharm.sh: [latest release](https://www.jetbrains.com/pycharm/download/other.html)
    - nteract.sh: [latest release](https://github.com/nteract/nteract/releases/latest)
    - r-runtime.sh: [latest release](https://www.rstudio.com/products/rstudio/download-server/)
-   - rstudio-server.sh: [latest release](https://www.rstudio.com/products/rstudio/download-server/)
-   - rstudio-desktop.sh: [latest release](https://www.rstudio.com/products/rstudio/download/#download)
    - sqlectron.sh: [latest release](https://github.com/sqlectron/sqlectron-gui/releases/latest)
    - zeppelin.sh: [latest release](http://zeppelin.apache.org/download.html)
    - robo3t.sh: [latest release](https://github.com/Studio3T/robomongo/releases/latest)
    - metabase.sh: [latest release](https://github.com/metabase/metabase/releases/latest)
    - fasttext.sh: [latest release](https://github.com/facebookresearch/fastText/releases/latest)
-   - kubernetes-utils.sh: [kube-prompt release](https://github.com/c-bata/kube-prompt/releases/latest), [conftest release](ttps://github.com/open-policy-agent/conftest), [yq release](https://github.com/mikefarah/yq/releases)
+   - kubernetes-utils.sh: [kube-prompt release](https://github.com/c-bata/kube-prompt/releases/latest), [conftest release](https://github.com/open-policy-agent/conftest/releases), [yq release](https://github.com/mikefarah/yq/releases)
    - portainer.sh: [latests release](https://github.com/portainer/portainer/releases/latest)
    - rapids-gpu.sh: [latests release](https://rapids.ai/)
 
-7. Update `minimmal` and `light` flavor python libraries:
+7. Update `minimmal` and `light` flavor Python libraries:
 
    - Update requirement files using [piprot](https://github.com/sesh/piprot), [pur](https://github.com/alanhamlett/pip-update-requirements), or [pip-upgrader](https://github.com/simion/pip-upgrader):
      - `piprot ./resources/libraries/requirements-minimal.txt`
@@ -67,7 +65,7 @@ We plan to do a full workspace image update (all libraries and tools) about ever
 
 8. Build and test `minimal` flavor:
 
-   - Build minimal workspace flavor via `python build.py --flavor=minimal`
+   - Build minimal workspace flavor via `python build.py --make --flavor=minimal`
    - Run workspace container and check startup logs
    - Check/Compare layer sizes of new image with previous version (via Portainer)
    - Check Image Labels (via Portainer)
@@ -79,16 +77,16 @@ We plan to do a full workspace image update (all libraries and tools) about ever
 
 9. Build and test `light` flavor:
 
-   - Build light workspace flavor via `python build.py --flavor=light`
+   - Build light workspace flavor via `python build.py --make --flavor=light`
    - Run workspace container and check startup logs
    - Check/Compare layer sizes of new image with previous version (via Portainer)
    - Check folder sizes via `Disk Usage Analyzer` within the Desktop VNC
-   - Run `/resources/tests/evaluate-python-libraries.ipynb` notebook to update `requirements-full.txt`
+   - Run `/resources/tests/evaluate-py-libraries.ipynb` notebook to update `requirements-full.txt`
    - Run `/resources/tests/test-tool-installers.ipynb` notebook to test installer scripts.
 
 10. Build and test `full` flavor:
 
-    - Build main workspace flavor via `python build.py --flavor=full`
+    - Build main workspace flavor via `python build.py --make --flavor=full`
     - Deploy new workspace image and check startup logs
     - Check/Compare layer sizes of new image with previous version (via Portainer)
     - Check Image Labels (via Portainer)
@@ -108,25 +106,12 @@ We plan to do a full workspace image update (all libraries and tools) about ever
 
 11. Update, build and test `gpu` flavor:
 
-   - Update CUDA Tooling based on [cuda container images](https://gitlab.com/nvidia/container-images/cuda/)
-   - Decide for CUDA version update based on tensorflow & pytorch support
-   - Update GPU libraries and tooling inside Dockerfile
-   - Build via `python build.py --flavor=gpu`
-   - Test `nvidia-smi` in terminal to check for GPU access
-   - Test image on GPU machine und run `/workspace/tutorials/workspace-test-utilities.ipynb`
-   - Test GPU interface in Netdata and Glances
+    - Update CUDA Tooling based on [cuda container images](https://gitlab.com/nvidia/container-images/cuda/)
+    - Decide for CUDA version update based on tensorflow & pytorch support
+    - Update GPU libraries and tooling inside Dockerfile
+    - Build via `python build.py --flavor=gpu`
+    - Test `nvidia-smi` in terminal to check for GPU access
+    - Test image on GPU machine und run `/workspace/tutorials/workspace-test-utilities.ipynb`
+    - Test GPU interface in Netdata and Glances
 
-12. Update, build and test `R` flavor:
-
-   - Build via `python build.py --flavor=R`
-   - Run `/workspace/tutorials/test-r-runtime.Rmd` via R kernel.
-   - Test `R Studio Server` tool and run the `/workspace/tutorials/test-r-runtime.Rmd`.
-
-13. Build and test `spark` flavor via `python build.py --flavor=spark`
-
-   - Build via `python build.py --flavor=spark`
-   - Run `/workspace/tutorials/test-spark.ipynb` via Python kernel.
-   - Run `/workspace/tutorials/toree-scala-kernel-tutorial.ipynb` via Toree kernel.
-   - Test `Zeppelin` tool.
-
-14. Build and push all flavors via `python build.py --deploy --version=<VERSION> --flavor=all`
+12. Build and push all flavors via `python build.py --deploy --version=<VERSION> --flavor=all`

docs/update-workspace-image.md

@@ -8,25 +8,27 @@ ENV WORKSPACE_FLAVOR=$ARG_WORKSPACE_FLAVOR
 USER root
 
 ### NVIDIA CUDA BASE ###
-# https://gitlab.com/nvidia/container-images/cuda/-/blob/master/dist/10.1/ubuntu18.04-x86_64/base/Dockerfile
-RUN apt-get update && apt-get install -y --no-install-recommends gnupg2 curl ca-certificates && \
-    curl -fsSL https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64/7fa2af80.pub | apt-key add - && \
-    echo "deb https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64 /" > /etc/apt/sources.list.d/cuda.list && \
-    echo "deb https://developer.download.nvidia.com/compute/machine-learning/repos/ubuntu1804/x86_64 /" > /etc/apt/sources.list.d/nvidia-ml.list && \
+# https://gitlab.com/nvidia/container-images/cuda/-/blob/master/dist/11.2.2/ubuntu20.04-x86_64/base/Dockerfile
+RUN apt-get update && apt-get install -y --no-install-recommends \
+    gnupg2 curl ca-certificates && \
+    curl -fsSL https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64/7fa2af80.pub | apt-key add - && \
+    echo "deb https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64 /" > /etc/apt/sources.list.d/cuda.list && \
+    echo "deb https://developer.download.nvidia.com/compute/machine-learning/repos/ubuntu2004/x86_64 /" > /etc/apt/sources.list.d/nvidia-ml.list && \
     # Cleanup - cannot use cleanup script here, otherwise too much is removed
     apt-get clean && \
     rm -rf $HOME/.cache/* && \
     rm -rf /tmp/* && \
     rm -rf /var/lib/apt/lists/*
 
-ENV CUDA_VERSION 10.1.243
-ENV CUDA_PKG_VERSION 10-1=$CUDA_VERSION-1
+ENV CUDA_VERSION 11.2.2
+#ENV CUDA_PKG_VERSION 11-2=$CUDA_VERSION-1
+#ENV CUDART_VERSION 11-2=$CUDA_VERSION46-1
 
 # For libraries in the cuda-compat-* package: https://docs.nvidia.com/cuda/eula/index.html#attachment-a
 RUN apt-get update && apt-get install -y --no-install-recommends \
-        cuda-cudart-$CUDA_PKG_VERSION \
-        cuda-compat-10-1 && \
-    ln -s cuda-10.1 /usr/local/cuda && \
+    cuda-cudart-11-2=11.2.152-1 \
+    cuda-compat-11-2 \
+    && ln -s cuda-11.2 /usr/local/cuda && \
     rm -rf /var/lib/apt/lists/* && \
     # Cleanup - cannot use cleanup script here, otherwise too much is removed
     apt-get clean && \
@@ -35,107 +37,101 @@ RUN apt-get update && apt-get install -y --no-install-recommends \
     rm -rf /var/lib/apt/lists/*
 
 # Required for nvidia-docker v1
-RUN echo "/usr/local/nvidia/lib" >> /etc/ld.so.conf.d/nvidia.conf && \
-    echo "/usr/local/nvidia/lib64" >> /etc/ld.so.conf.d/nvidia.conf
+RUN echo "/usr/local/nvidia/lib" >> /etc/ld.so.conf.d/nvidia.conf \
+    && echo "/usr/local/nvidia/lib64" >> /etc/ld.so.conf.d/nvidia.conf
 
 ENV PATH /usr/local/nvidia/bin:/usr/local/cuda/bin:${PATH}
-ENV LD_LIBRARY_PATH /usr/local/nvidia/lib:/usr/local/nvidia/lib64:${LD_LIBRARY_PATH}
+ENV LD_LIBRARY_PATH /usr/local/nvidia/lib:/usr/local/nvidia/lib64
 
 # nvidia-container-runtime
 # https://github.com/NVIDIA/nvidia-container-runtime#environment-variables-oci-spec
 # nvidia-container-runtime
 ENV NVIDIA_VISIBLE_DEVICES all
 ENV NVIDIA_DRIVER_CAPABILITIES compute,utility
-ENV NVIDIA_REQUIRE_CUDA "cuda>=10.1 brand=tesla,driver>=396,driver<397 brand=tesla,driver>=410,driver<411 brand=tesla,driver>=418,driver<419"
+ENV NVIDIA_REQUIRE_CUDA "cuda>=11.2 brand=tesla,driver>=418,driver<419 brand=tesla,driver>=440,driver<441 driver>=450"
 
 ### CUDA RUNTIME ###
-# https://gitlab.com/nvidia/container-images/cuda/-/blob/master/dist/10.1/ubuntu18.04-x86_64/runtime/Dockerfile
+# https://gitlab.com/nvidia/container-images/cuda/-/blob/master/dist/11.2.2/ubuntu20.04-x86_64/runtime/Dockerfile
 
-ENV NCCL_VERSION 2.7.8
+ENV NCCL_VERSION 2.8.4
 
 RUN apt-get update && apt-get install -y --no-install-recommends \
-        cuda-libraries-$CUDA_PKG_VERSION \
-        cuda-npp-$CUDA_PKG_VERSION \
-        cuda-nvtx-$CUDA_PKG_VERSION \
-        libcublas10=10.2.1.243-1 \
-        libnccl2=$NCCL_VERSION-1+cuda10.1 && \
-    apt-mark hold libnccl2 && \
+    cuda-libraries-11-2=11.2.2-1 \
+    libnpp-11-2=11.3.2.152-1 \
+    cuda-nvtx-11-2=11.2.152-1 \
+    libcublas-11-2=11.4.1.1043-1 \
+    libcusparse-11-2=11.4.1.1152-1 \
+    libnccl2=$NCCL_VERSION-1+cuda11.2 \
+    && rm -rf /var/lib/apt/lists/* \
     # Cleanup - cannot use cleanup script here, otherwise too much is removed
-    apt-get clean && \
-    rm -rf $HOME/.cache/* && \
-    rm -rf /tmp/* && \
-    rm -rf /var/lib/apt/lists/*
+    && apt-get clean \
+    && rm -rf $HOME/.cache/* \
+    && rm -rf /tmp/* \
+    && rm -rf /var/lib/apt/lists/*
 
-# apt from auto upgrading the cublas package. See https://gitlab.com/nvidia/container-images/cuda/-/issues/88
-RUN apt-mark hold libcublas10
+RUN apt-mark hold libcublas-11-2 libnccl2
 
 ### END CUDA RUNTIME ###
 
 ### CUDA DEVEL ###
-# https://gitlab.com/nvidia/container-images/cuda/-/blob/master/dist/10.1/ubuntu18.04-x86_64/devel/Dockerfile
+# https://gitlab.com/nvidia/container-images/cuda/-/blob/master/dist/11.2.2/ubuntu20.04-x86_64/devel/Dockerfile
 RUN apt-get update && apt-get install -y --no-install-recommends \
-        cuda-nvml-dev-$CUDA_PKG_VERSION \
-        cuda-command-line-tools-$CUDA_PKG_VERSION \
-        cuda-nvprof-$CUDA_PKG_VERSION \
-        cuda-npp-dev-$CUDA_PKG_VERSION \
-        cuda-libraries-dev-$CUDA_PKG_VERSION \
-        cuda-minimal-build-$CUDA_PKG_VERSION \
-        libcublas-dev=10.2.1.243-1 \
-        libnccl-dev=$NCCL_VERSION-1+cuda10.1 && \
-    apt-mark hold libnccl-dev && \
+    libtinfo5 libncursesw5 \
+    cuda-cudart-dev-11-2=11.2.152-1 \
+    cuda-command-line-tools-11-2=11.2.2-1 \
+    cuda-minimal-build-11-2=11.2.2-1 \
+    cuda-libraries-dev-11-2=11.2.2-1 \
+    cuda-nvml-dev-11-2=11.2.152-1 \
+    libnpp-dev-11-2=11.3.2.152-1 \
+    libnccl-dev=2.8.4-1+cuda11.2 \
+    libcublas-dev-11-2=11.4.1.1043-1 \
+    libcusparse-dev-11-2=11.4.1.1152-1 && \
     # Cleanup - cannot use cleanup script here, otherwise too much is removed
     apt-get clean && \
     rm -rf $HOME/.cache/* && \
     rm -rf /tmp/* && \
     rm -rf /var/lib/apt/lists/*
 
 # apt from auto upgrading the cublas package. See https://gitlab.com/nvidia/container-images/cuda/-/issues/88
-RUN apt-mark hold libcublas-dev
-
+RUN apt-mark hold libcublas-dev-11-2 libnccl-dev
 ENV LIBRARY_PATH /usr/local/cuda/lib64/stubs
 
 ### END CUDA DEVEL ###
 
-### CUDANN7 DEVEL ###
-# https://gitlab.com/nvidia/container-images/cuda/-/blob/master/dist/10.1/ubuntu18.04-x86_64/devel/cudnn7/Dockerfile
+### CUDANN8 DEVEL ###
+# https://gitlab.com/nvidia/container-images/cuda/-/blob/master/dist/11.2.2/ubuntu20.04-x86_64/devel/cudnn8/Dockerfile
 
-ENV CUDNN_VERSION 7.6.5.32
+ENV CUDNN_VERSION 8.1.1.33
 LABEL com.nvidia.cudnn.version="${CUDNN_VERSION}"
 
-RUN apt-get update && \
-    apt-get install -y --no-install-recommends \
-            libcudnn7=$CUDNN_VERSION-1+cuda10.1 \
-            libcudnn7-dev=$CUDNN_VERSION-1+cuda10.1 && \
-    apt-mark hold libcudnn7 && \
+RUN apt-get update && apt-get install -y --no-install-recommends \
+    libcudnn8=$CUDNN_VERSION-1+cuda11.2 \
+    libcudnn8-dev=$CUDNN_VERSION-1+cuda11.2 \
+    && apt-mark hold libcudnn8 && \
     # Cleanup
     apt-get clean && \
     rm -rf /root/.cache/* && \
     rm -rf /tmp/* && \
     rm -rf /var/lib/apt/lists/*
 
-### END CUDANN7 ###
+### END CUDANN8 ###
 
 # Link Cupti:
 ENV LD_LIBRARY_PATH ${LD_LIBRARY_PATH}:/usr/local/cuda/extras/CUPTI/lib64
 
-# Install TensorRT. Requires that libcudnn7 is installed above.
-# https://www.tensorflow.org/install/gpu#ubuntu_1804_cuda_101
-RUN apt-get update && apt-get install -y --no-install-recommends \
-        libnvinfer6=6.0.1-1+cuda10.1 \
-        libnvinfer-dev=6.0.1-1+cuda10.1 \
-        libnvinfer-plugin6=6.0.1-1+cuda10.1 && \
-    # Cleanup
-    clean-layer.sh
-
 ### GPU DATA SCIENCE LIBRARIES ###
 
 RUN \
     apt-get update && \
     apt-get install -y libomp-dev libopenblas-base && \
-    # Not needed? Install cuda-toolkit (e.g. for pytorch: https://pytorch.org/): https://anaconda.org/anaconda/cudatoolkit
-    conda install -y cudatoolkit=10.1 -c pytorch && \
+    # Install pytorch gpu
+    # uninstall cpu only packages via conda
+    conda remove --force -y pytorch cpuonly && \
+    # https://pytorch.org/get-started/locally/
+    conda install cudatoolkit=11.2 -c pytorch -c nvidia && \
+    pip install --no-cache-dir torch==1.9.0+cu111 torchvision==0.10.0+cu111 torchaudio==0.9.0 -f https://download.pytorch.org/whl/torch_stable.html && \
     # Install cupy: https://cupy.chainer.org/
-    pip install --no-cache-dir cupy-cuda101 && \
+    pip install --no-cache-dir cupy-cuda112 && \
     # Install pycuda: https://pypi.org/project/pycuda
     pip install --no-cache-dir pycuda && \
     # Install gpu utils libs
@@ -144,25 +140,19 @@ RUN \
     pip install --no-cache-dir scikit-cuda && \
     # Install tensorflow gpu
     pip uninstall -y tensorflow tensorflow-cpu intel-tensorflow && \
-    # TODO: tensorflow 2.3.1 installs tenorboard 2.4.0 with problems, use 2.3.0
-    pip install --no-cache-dir tensorflow-gpu==2.3.0 && \
+    pip install --no-cache-dir tensorflow-gpu==2.5.0 && \
     # Install ONNX GPU Runtime
-    # TODO: 1.4.x is latest with cuda 10.1 support
     pip uninstall -y onnxruntime && \
-    pip install --no-cache-dir onnxruntime-gpu==1.4.0 && \
-    # Install pytorch gpu
-    # uninstall cpu only packages via conda
-    conda remove --force -y pytorch cpuonly && \
-    # https://pytorch.org/get-started/locally/
-    conda install -y pytorch -c pytorch && \
-    # Install faiss gpu
-    conda remove --force -y faiss-cpu && \
-    conda install -y faiss-gpu -c pytorch && \
+    pip install --no-cache-dir onnxruntime-gpu==1.8.0 onnxruntime-training==1.8.0 && \
+    # Install faiss gpu - TODO: to large?
+    # conda remove --force -y faiss-cpu && \
+    # conda install -y faiss-gpu -c pytorch && \
     # Update mxnet to gpu edition
     pip uninstall -y mxnet-mkl && \
-    pip install --no-cache-dir mxnet-cu101mkl==1.6.0.post0 && \
+    # cuda111 -> >= 11.1
+    pip install --no-cache-dir mxnet-cu112 && \
     # install jax: https://github.com/google/jax#pip-installation
-    pip install --upgrade jax jaxlib==0.1.57+cuda101 -f https://storage.googleapis.com/jax-releases/jax_releases.html  && \
+    pip install --upgrade jax[cuda111] -f https://storage.googleapis.com/jax-releases/jax_releases.html && \
     # Install pygpu - Required for theano: http://deeplearning.net/software/libgpuarray/
     conda install -y pygpu && \
     # Install lightgbm
@@ -177,19 +167,6 @@ RUN \
     # Cleanup
     clean-layer.sh
 
-# TODO: nvdashboard does not work with relative paths
-# RUN \
-#     # Install Jupyterlab GPU Plugin: https://github.com/rapidsai/jupyterlab-nvdashboard
-#     pip install jupyterlab-nvdashboard && \
-#     jupyter labextension install jupyterlab-nvdashboard && \
-#     # Clean jupyter lab cache: https://github.com/jupyterlab/jupyterlab/issues/4930
-#     jupyter lab clean && \
-#     jlpm cache clean && \
-#     # Remove build folder -> should be remove by lab clean as well?
-#     rm -rf $CONDA_ROOT/share/jupyter/lab/staging && \
-#     # Cleanup
-#     clean-layer.sh
-
 # TODO install DALI: https://docs.nvidia.com/deeplearning/dali/user-guide/docs/installation.html#dali-and-ngc
 # TODO: if > Ubuntu 19.04 -> install nvtop: https://github.com/Syllo/nvtop
 # TODO: Install Arrrayfire: https://arrayfire.com/download/ pip install --no-cache-dir arrayfire && \