azure: first attempt to build base container (#5)

* azure: first attempt to build base container

This is entirely automated. We use the assets from their azurevm
hpc repository, and add flux. Importantly, the environment for
hpcx needs to be sourced, and likely we will need to do that
in different environments (kubernetes/usernetes/singularity)

Signed-off-by: vsoch <[email protected]>

Author: vsoch

tutorial/azure/README.md

@@ -107,29 +107,26 @@ done
 pssh -h hosts.txt -x "-i ./id_azure" "/bin/bash /tmp/update_brokers.sh flux $lead_broker"
 ```
 
-Note that I've also provided a script to install the OSU benchmarks with the same strategy above:
+Note that I've also provided scripts to install the OSU benchmarks and lammps with the same strategy above:
 
 ```bash
-for address in $(az vmss list-instance-public-ips -g terraform-testing -n flux | jq -r .[].ipAddress)
- do
-   echo "Updating $address"
-   scp -i ./id_azure install_osu.sh azureuser@${address}:/tmp/install_osu.sh
-done
-pssh -h hosts.txt -x "-i ./id_azure" "/bin/bash /tmp/install_osu.sh flux $lead_broker"
+# Choose the script you want to install
+script=install_osu.sh
+script=install_lammps.sh
 ```
 
-This installs to `/usr/local/libexec/osu-benchmarks/mpi`. And lammps:
+And then install!
 
-```bash
+```console
 for address in $(az vmss list-instance-public-ips -g terraform-testing -n flux | jq -r .[].ipAddress)
  do
    echo "Updating $address"
-   scp -i ./id_azure install_lammps.sh azureuser@${address}:/tmp/install_lammps.sh
+   scp -i ./id_azure ./install/${script} azureuser@${address}:/tmp/${script}
 done
-pssh -h hosts.txt -x "-i ./id_azure" "/bin/bash /tmp/install_lammps.sh flux $lead_broker"
+pssh -h hosts.txt -x "-i ./id_azure" "/bin/bash /tmp/${script}"
 ```
-That installs to `/usr/bin/lmp`
 
+This installs to `/usr/local/libexec/osu-micro-benchmarks/mpi`. And lammps installs to `/usr/bin/lmp`
 
 ### 3. Checks
 
@@ -142,7 +139,7 @@ flux resource list
 flux run -N 2 hostname
 ```
 
-### 4. Benchmarks
+### 4. Applications and Benchmarks
 
 Try running a benchmark!
 
@@ -278,6 +275,10 @@ Total wall time: 0:00:37
 
 </details>
 
+#### Usernetes
+
+See [flux-usernetes](https://github.com/converged-computing/flux-usernetes/tree/main/azure) for build and deploy instructions for deployment of user space kubernetes.
+
 ### 4. Cleanup
 
 This should work (but see [debugging](#debugging)).
@@ -840,6 +841,10 @@ This is free software; see the source for copying conditions.  There is NO
 warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
 ```
 
+### Docker
+
+For advanced users, we have a [docker](docker) directory with builds that emulate the base set of VMs that are intended to be used with them. It would be good if Microsoft wanted to provide more production bases for us :)
+
 ### Debugging
 
 Depending on your environment, terraform (e.g., `make` or `make destroy` doesn't always work. I get this error from the Azure Cloud Shell:

tutorial/azure/docker/README.md

@@ -0,0 +1,29 @@
+# Azure Docker Builds
+
+We hope that Microsoft can eventually provide container bases, but until then we need to make a best effort to do that. This attempts to mirror the logic and match versions for their Azure HPC images builds.
+
+## Base
+
+The base image has core dependencies like hpcx and flux.
+
+```bash
+cd ./base
+docker build -t ghcr.io/converged-computing/flux-tutorials:azurehpc-2204 .
+docker push ghcr.io/converged-computing/flux-tutorials:azurehpc-2204
+```
+
+## OSU
+
+```bash
+cd ./osu
+docker build -t ghcr.io/converged-computing/flux-tutorials:azurehpc-2204-osu .
+docker push ghcr.io/converged-computing/metric-osu-cpu:azure-hpc-osu
+```
+
+## LAMMPS
+
+```bash
+cd ./lammps-reax
+docker build -t ghcr.io/converged-computing/flux-tutorials:azurehpc-2204-lammps-reax .
+docker push ghcr.io/converged-computing/metric-osu-cpu:azure-hpc-lammps-reax
+```

tutorial/azure/docker/base/Dockerfile

@@ -0,0 +1,205 @@
+FROM ubuntu:22.04
+
+# docker build -t ghcr.io/converged-computing/flux-tutorials:azurehpc-2204 .
+# docker push ghcr.io/converged-computing/flux-tutorials:azurehpc-2204
+
+WORKDIR /opt
+RUN apt-get update && apt-get install -y munge git curl wget unzip gpg debian-archive-keyring \  
+    pkg-config vim ubuntu-keyring systemctl && apt-get clean
+RUN export VERSION="1.2.2" && \
+    curl -LO "https://github.com/oras-project/oras/releases/download/v${VERSION}/oras_${VERSION}_linux_amd64.tar.gz" && \
+    mkdir -p oras-install/ && \
+    tar -zxf oras_${VERSION}_*.tar.gz -C oras-install/ && \
+    mv oras-install/oras /usr/local/bin/ && \
+    rm -rf oras_${VERSION}_*.tar.gz oras-install/
+
+# Azure hpc-images deps added here - not clear if all of these are needed
+RUN apt-get update && apt-get install -y numactl rpm libnuma-dev libmpc-dev libmpfr-dev libxml2-dev m4 byacc \
+         libnl-3-dev libnl-route-3-dev libnl-3-200 libnl-genl-3-dev libnl-genl-3-200 libnl-route-3-200 bison \
+         libsecret-1-0 dkms libyaml-dev libreadline-dev libkeyutils1 libkeyutils-dev libmount-dev nfs-common pssh \
+         libvulkan1 hwloc selinux-policy-dev nvme-cli && apt-get clean # vulkan is for nvidia gpu driver
+ENV DEBIAN_FRONTEND=noninteractive
+
+# OSU Benchmarks in hpcx-v2.19-gcc-mlnx_ofed-ubuntu22.04-cuda12-x86_64/ompi/tests/
+RUN oras pull ghcr.io/converged-computing/rdma-infiniband:ubuntu-22.04-tgz --output /opt && \
+    cd /opt && \
+    tar -xzvf MLNX_OFED_LINUX-24.04-0.7.0.0-ubuntu22.04-x86_64.tgz && \
+    cd MLNX_OFED_LINUX-24.04-0.7.0.0-ubuntu22.04-x86_64/DEBS/ && \
+    dpkg -i mpitests_3.2.23-45a045b.2404066_amd64.deb && \
+    dpkg -i libibverbs1* && \
+    dpkg -i ibverbs-providers* && \
+    dpkg -i libibverbs* && \
+    dpkg -i librdmacm* && \
+    dpkg -i ucx_1.17.0-1.2404066_amd64.deb && \
+    dpkg -i libibumad3* && \
+    dpkg -i sharp_3.7.0.MLNX20240421.48444036-1.2404066_amd64.deb && \
+    dpkg -i hcoll_4.8.3227-1.2404066_amd64.deb 
+
+# This was extracted into separate lines, below, to avoid one large layer (and debug each)
+# RUN ./install.sh
+ENV GPU=NVIDIA
+
+# Install only what we need as we go (so change to single file doesn't require complete rebuild)
+WORKDIR /opt/azhpc-images/ubuntu/ubuntu-22.x/ubuntu-22.04-hpc
+COPY ./azhpc-images/ubuntu/ubuntu-22.x/ubuntu-22.04-hpc/install_prerequisites.sh ./install_prerequisites.sh
+
+# install pre-requisites
+RUN ./install_prerequisites.sh
+COPY ./azhpc-images/versions.json /opt/azhpc-images/versions.json
+
+COPY ./azhpc-images/ubuntu/ubuntu-22.x/ubuntu-22.04-hpc/set_properties.sh ./
+COPY ./azhpc-images/ubuntu/ubuntu-22.x/ubuntu-22.04-hpc/install_utils.sh ./
+COPY ./azhpc-images/ubuntu/common/remove_unused_packages.sh /opt/azhpc-images/ubuntu/common/remove_unused_packages.sh
+COPY ./azhpc-images/ubuntu/common/install_utils.sh /opt/azhpc-images/ubuntu/common/install_utils.sh
+COPY ./azhpc-images/ubuntu/common/install_pmix.sh /opt/azhpc-images/ubuntu/common/install_pmix.sh
+COPY ./azhpc-images/ubuntu/common/install_pmix.sh /opt/azhpc-images/ubuntu/common/install_mpis.sh
+COPY ./azhpc-images/common/ /opt/azhpc-images/common/
+COPY ./azhpc-images/tools/ /opt/azhpc-images/tools/
+
+# remove packages requiring Ubuntu Pro for security updates
+RUN . ./set_properties.sh && \
+    /bin/bash $UBUNTU_COMMON_DIR/remove_unused_packages.sh && \
+    ./install_utils.sh
+
+COPY ./azhpc-images/ubuntu/common/install_docker.sh /opt/azhpc-images/ubuntu/common/install_docker.sh
+COPY ./azhpc-images/ubuntu/common/* /opt/azhpc-images/ubuntu/common/
+RUN . ./set_properties.sh && \
+    /bin/bash $UBUNTU_COMMON_DIR/install_docker.sh
+
+# install diagnostic script, optimizations
+RUN . ./set_properties.sh && \
+    /bin/bash $COMMON_DIR/install_hpcdiag.sh && \
+    /bin/bash $COMMON_DIR/install_azure_persistent_rdma_naming.sh
+
+RUN . ./set_properties.sh && \
+    /bin/bash $UBUNTU_COMMON_DIR/hpc-tuning.sh
+
+COPY ./azhpc-images/tests/ /opt/azhpc-images/tests
+COPY ./azhpc-images/customizations/ /opt/azhpc-images/customizations
+COPY ./azhpc-images/topology/ /opt/azhpc-images/topology
+
+RUN . ./set_properties.sh && \
+    /bin/bash $COMMON_DIR/copy_test_file.sh && \
+    /bin/bash $COMMON_DIR/install_monitoring_tools.sh && \
+    /bin/bash $COMMON_DIR/install_amd_libs.sh
+
+RUN . ./set_properties.sh && \
+    /bin/bash $COMMON_DIR/setup_sku_customizations.sh
+
+RUN . ./set_properties.sh && \
+    /bin/bash $UBUNTU_COMMON_DIR/install_pmix.sh
+
+# For some reason this command, when moved higher up, was flaky. 
+# Watch it and make sure it doesn't skip (if it does the build will fail later)
+RUN . ./set_properties.sh && \
+    /bin/bash $UBUNTU_COMMON_DIR/install_mpis.sh
+
+# This would match the VM exactly (you'd ned to change the source script, etc).
+# RUN mv /opt/hpcx-v2.19-gcc-mlnx_ofed-ubuntu22.04-cuda12-x86_64 /opt/hpcx-v2.15-gcc-MLNX_OFED_LINUX-5-ubuntu22.04-cuda12-gdrcopy2-nccl2.17-x86_64/
+
+# cleanup downloaded tarballs - clear some space
+RUN rm -rf *.tgz *.bz2 *.tbz *.tar.gz *.run *.deb *_offline.sh && \
+    rm -rf /tmp/MLNX_OFED_LINUX* /tmp/*conf* && \
+    rm -rf /var/intel/ /var/cache/* && \
+    rm -Rf -- */
+
+# INFO: Building OMPI with HCOLL
+# Ready to rebuild
+#  HPCX_ROOT:      /opt/hpcx-v2.19-gcc-mlnx_ofed-ubuntu22.04-cuda12-x86_64
+#  OMPI PREFIX:    /opt/hpcx-v2.19-gcc-mlnx_ofed-ubuntu22.04-cuda12-x86_64/hpcx-rebuild
+#  UCX location:   /opt/hpcx-v2.19-gcc-mlnx_ofed-ubuntu22.04-cuda12-x86_64/ucx
+#  UCC location:   /opt/hpcx-v2.19-gcc-mlnx_ofed-ubuntu22.04-cuda12-x86_64/ucc
+#  HCOLL location: /opt/hpcx-v2.19-gcc-mlnx_ofed-ubuntu22.04-cuda12-x86_64/hcoll
+
+ENV LANG=C.UTF-8
+RUN apt-get update && apt-get install -y locales && locale-gen en_US.UTF-8 && apt-get clean
+
+# Add an azureuser
+ARG USER=azureuser
+ARG UID=1000
+ARG GID=1000
+RUN set -x && groupadd -g $UID $USER && \
+    useradd -g $USER -u $UID -d /home/$USER -m $USER && \
+    printf "$USER ALL= NOPASSWD: ALL\\n" >> /etc/sudoers
+
+# flux security
+WORKDIR /opt/flux
+
+RUN apt-get update && \
+    apt-get install -y man flex ssh sudo vim luarocks munge lcov ccache lua5.4 \
+         valgrind build-essential pkg-config autotools-dev libtool \
+         libffi-dev autoconf automake make clang clang-tidy \
+         gcc g++ libpam-dev apt-utils lua-posix \
+         libsodium-dev libzmq3-dev libczmq-dev libjansson-dev libmunge-dev \
+         libncursesw5-dev liblua5.4-dev liblz4-dev libsqlite3-dev uuid-dev \
+         libhwloc-dev libs3-dev libevent-dev libarchive-dev \
+         libboost-graph-dev libboost-system-dev libboost-filesystem-dev \
+         libboost-regex-dev libyaml-cpp-dev libedit-dev uidmap dbus-user-session python3-cffi && apt-get clean
+
+COPY ./azhpc-images/source-hpcx.sh /source-hpcx.sh
+RUN . /source-hpcx.sh && hpcx_load && \
+    wget https://github.com/flux-framework/flux-security/releases/download/v0.13.0/flux-security-0.13.0.tar.gz && \
+    tar -xzvf flux-security-0.13.0.tar.gz && \
+    mv flux-security-0.13.0 /opt/flux/flux-security && \
+    cd /opt/flux/flux-security && \
+    ./configure --prefix=/usr --sysconfdir=/etc && \
+    make -j && make install
+
+# The VMs will share the same munge key
+RUN mkdir -p /var/run/munge && \
+    dd if=/dev/urandom bs=1 count=1024 > munge.key && \
+    mv munge.key /etc/munge/munge.key && \
+    chown -R munge /etc/munge/munge.key /var/run/munge && \
+    chmod 600 /etc/munge/munge.key
+
+# Make the flux run directory
+RUN mkdir -p /home/azureuser/run/flux && chown azureuser /home/azureuser
+RUN python3 -m pip install jsonschema --upgrade
+
+# Flux core
+RUN . /source-hpcx.sh && hpcx_load && \
+    wget https://github.com/flux-framework/flux-core/releases/download/v0.68.0/flux-core-0.68.0.tar.gz && \
+    tar -xzvf flux-core-0.68.0.tar.gz && \
+    mv flux-core-0.68.0 /opt/flux/flux-core && \
+    cd /opt/flux/flux-core && \
+    ./configure --prefix=/usr --sysconfdir=/etc --with-flux-security && \
+    make clean && \
+    make -j && make install
+
+# Flux sched (later than this requires newer gcc and clang)
+RUN . /source-hpcx.sh && hpcx_load && \
+    wget https://github.com/flux-framework/flux-sched/releases/download/v0.37.0/flux-sched-0.37.0.tar.gz && \
+    tar -xzvf flux-sched-0.37.0.tar.gz && \
+    mv flux-sched-0.37.0 /opt/flux/flux-sched && \
+    cd /opt/flux/flux-sched && \
+    mkdir build && \
+    cd build && \
+    cmake ../ && make -j && make install && ldconfig && \
+    echo "DONE flux build"
+
+# Flux curve.cert
+# Ensure we have a shared curve certificate
+RUN flux keygen /tmp/curve.cert && \
+    mkdir -p /etc/flux/system && \
+    cp /tmp/curve.cert /etc/flux/system/curve.cert && \
+    chown azureuser /etc/flux/system/curve.cert && \
+    chmod o-r /etc/flux/system/curve.cert && \
+    chmod g-r /etc/flux/system/curve.cert && \
+    # Permissions for imp
+    chmod u+s /usr/libexec/flux/flux-imp && \
+    chmod 4755 /usr/libexec/flux/flux-imp && \
+    # /var/lib/flux needs to be owned by the instance owner
+    mkdir -p /var/lib/flux && \
+    chown azureuser -R /var/lib/flux && \
+    # clean up (and make space)
+    cd /opt && \
+    rm -rf /opt/flux
+
+# Ensure we source the environment.
+RUN echo ". /opt/hpcx-v2.15-gcc-MLNX_OFED_LINUX-5-ubuntu22.04-cuda12-gdrcopy2-nccl2.17-x86_64/hpcx-mt-init.sh" >> /root/.bashrc && \
+    echo ". /opt/hpcx-v2.15-gcc-MLNX_OFED_LINUX-5-ubuntu22.04-cuda12-gdrcopy2-nccl2.17-x86_64/hpcx-mt-init.sh" >> /home/azureuser/.bashrc && \
+    echo "hpcx_load" >> /root/.bashrc && \
+    echo "hpcx_load" >> /home/azureuser/.bashrc && \
+    echo "FLUX_URI        DEFAULT=local:///opt/run/flux/local" >> ./environment && \
+    mv ./environment /etc/security/pam_env.conf
+WORKDIR /opt

tutorial/azure/docker/base/Dockerfile.ucx

@@ -0,0 +1,18 @@
+FROM ghcr.io/converged-computing/flux-tutorials:azurehpc-2204
+
+# Note that this isn't currently built or used, as ucx is provided in the base
+# However, we are anticipating a bug we had earlier with needing to build ucx with
+# a different flag, and preserving this example for that.
+
+# Let's compile UCX without GPU checking.
+# This is the one we wound up using for our experiments
+
+# docker build -f Dockerfile.ucx -t ghcr.io/converged-computing/azurehpc:flux-slim-nogpu .
+# docker push ghcr.io/converged-computing/azurehpc:flux-slim-nogpu
+
+# Get build flags with ucx_info -b
+RUN wget https://github.com/openucx/ucx/releases/download/v1.15.0/ucx-1.15.0.tar.gz && \
+    tar -xzvf ucx-1.15.0.tar.gz && \
+    cd ucx-1.15.0 && \
+    ./configure --disable-logging --disable-debug --disable-assertions --disable-params-check --enable-mt --prefix=/usr --enable-examples --without-java --without-go --without-xpmem --without-cuda && \
+    make -j4 && make install && ldconfig

tutorial/azure/docker/base/azhpc-images/.gitattributes

@@ -0,0 +1 @@
+*.* eol=lf

tutorial/azure/docker/base/azhpc-images/LICENSE

@@ -0,0 +1,21 @@
+    MIT License
+
+    Copyright (c) Microsoft Corporation.
+
+    Permission is hereby granted, free of charge, to any person obtaining a copy
+    of this software and associated documentation files (the "Software"), to deal
+    in the Software without restriction, including without limitation the rights
+    to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+    copies of the Software, and to permit persons to whom the Software is
+    furnished to do so, subject to the following conditions:
+
+    The above copyright notice and this permission notice shall be included in all
+    copies or substantial portions of the Software.
+
+    THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+    IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+    FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+    AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+    LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+    OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+    SOFTWARE

tutorial/azure/docker/base/azhpc-images/azure-pipelines.yml

@@ -0,0 +1,25 @@
+trigger: none
+
+pr:
+- master
+
+pool: 
+  name: 1ES-hosted-pool-scrub1
+
+jobs:
+- job: queue_azdo
+  timeoutInMinutes: 360
+  steps:
+  - bash: |
+      echo $(System.PullRequest.PullRequestNumber)
+    displayName: Print PR Num
+
+  - task: Bash@3
+    inputs:
+      targetType: 'filePath'
+      filePath: './azure-pipelines/queue_ado.sh'
+      failOnStderr: true
+    env:
+      SYSTEM_ACCESSTOKEN: $(System.AccessToken)
+      PR_NUM: $(System.PullRequest.PullRequestNumber)
+    displayName: Queue Validation Build and Monitor Status