From 18a07ffdae99dbcf7199094ccea5fdb891a5c8e6 Mon Sep 17 00:00:00 2001 From: gwangmu <43980405+gwangmu@users.noreply.github.com> Date: Tue, 25 Feb 2025 11:16:23 +0100 Subject: [PATCH 01/34] Create container-engine.md --- docs/tools/container-engine.md | 897 +++++++++++++++++++++++++++++++++ 1 file changed, 897 insertions(+) create mode 100644 docs/tools/container-engine.md diff --git a/docs/tools/container-engine.md b/docs/tools/container-engine.md new file mode 100644 index 00000000..bbeb252e --- /dev/null +++ b/docs/tools/container-engine.md @@ -0,0 +1,897 @@ +[](){#container-engine} +# Container Engine + +The Container Engine (CE) toolset is designed to enable computing jobs to seamlessly run inside Linux application containers, thus providing support for containerized user environments. + +## Concept + +Containers effectively encapsulate a software stack; however, to be useful in HPC computing environments, they often require the customization of bind mounts, environment variables, working directories, hooks, plugins, etc. To simplify this process, the Container Engine (CE) toolset supports the specification of user environments through Environment Definition Files. + +An Environment Definition File (EDF) is a text file in the TOML format that declaratively and prescriptively represents the creation of a computing environment based on a container image. Users can create their own custom environments and share, edit, or build upon already existing environments. + +The Container Engine (CE) toolset leverages its tight integration with the Slurm workload manager to parse EDFs directly from the command line or batch script and instantiate containerized user environments seamlessly and transparently. + +Through the EDF, container use cases can be abstracted to the point where end users perform their workflows as if they were operating natively on the computing system. + +**Key Benefits** + * *Freedom*: Container gives users full control of the user space. The user can decide what to install without involving a sysadmin. + * *Reproducibility*: Workloads consistently run in the same environment, ensuring uniformity across job experimental runs. + * *Portability*: The self-contained nature of containers simplifies the deployment across architecture-compatible HPC systems. + * *Seamless Access to HPC Resources*: CE facilitates native access to specialized HPC resources like GPUs, interconnects, and other system-specific tools crucial for performance + +## Quick Start + +Let's set up a containerized Ubuntu 24.04 environment using a scratch folder as the working directory. + +### Example EDF + +``` +image = "library/ubuntu:24.04" +mounts = ["/capstor/scratch/cscs/${USER}:/capstor/scratch/cscs/${USER}"] +workdir = "/capstor/scratch/cscs/${USER}" +``` + +Note: Ensure that your ${USER} environment variable is defined with your actual username. + +Save this file as ubuntu.toml file in $HOME/.edf directory (which is the default location of EDF files). A more detailed explanation of each entry for the EDF can be seen in the EDF reference. + +### Running the environment + +Use Slurm in the cluster login node to start the Ubuntu environment that was just defined as follows: + +``` +$ srun --environment=ubuntu --pty bash +``` + +Since the ubuntu.toml file is located in the EDF search path, the filename can be passed to the option without the file extension. + +### Example Output + +```bash +[][@-ln001 ~]$ srun --environment=ubuntu --pty bash # Step 1 + +@:/capstor/scratch/cscs/$ pwd # Step 2 +/capstor/scratch/cscs/ + +@:/capstor/scratch/cscs/$ cat /etc/os-release # Step 3 +PRETTY_NAME="Ubuntu 24.04 LTS" +NAME="Ubuntu" +VERSION_ID="24.04" +VERSION="24.04 LTS (Noble Numbat)" +VERSION_CODENAME=noble +ID=ubuntu +ID_LIKE=debian +HOME_URL="https://www.ubuntu.com/" +SUPPORT_URL="https://help.ubuntu.com/" +BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" +PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" +UBUNTU_CODENAME=noble +LOGO=ubuntu-logo + +@:/capstor/scratch/cscs/$ exit # Step 4 +[][@-ln001 ~]$ +``` + +The above terminal snippet demonstrates how to launch a containerized environment using Slurm with the --environment option, where we highlight: + + Step 1. Starting an interactive shell session within the Ubuntu 24.04 container deployed on a compute node using srun --environment=ubuntu --pty bash. + Step 2. Confirm the working directory inside the container (pwd) and that it is set to the user's scratch folder, as per EDF. + Step 3. Show the OS version of your container (using cat /etc/os-release) based on Ubuntu 24.04 LTS. + Step 4. Exiting the container (exit), returning to the login node. + +Note that the image pull and the container start happen automatically, streamlining the usage of the CE. + +## Running containerized environments + +A job is run in a containerized environment by passing the --environment option to the srun or salloc Slurm commands. The option takes a file path to the EDF describing the environment in which the job should be executed, for example: + +```bash +[][@-ln001 ~]$ srun --environment=$SCRATCH/edf/debian.toml cat /etc/os-release +PRETTY_NAME="Debian GNU/Linux 12 (bookworm)" +NAME="Debian GNU/Linux" +VERSION_ID="12" +VERSION="12 (bookworm)" +VERSION_CODENAME=bookworm +ID=debian +HOME_URL="https://www.debian.org/" +SUPPORT_URL="https://www.debian.org/support" +BUG_REPORT_URL="https://bugs.debian.org/" + +--environment can be a relative path from the current working directory (i.e., where the Slurm command is entered). A relative path should be prepended by ./. For example: + +[][@-ln001 ~]$ ls +debian.toml +[][@-ln001 ~]$ srun --environment=./debian.toml cat /etc/os-release +PRETTY_NAME="Debian GNU/Linux 12 (bookworm)" +NAME="Debian GNU/Linux" +VERSION_ID="12" +VERSION="12 (bookworm)" +VERSION_CODENAME=bookworm +ID=debian +HOME_URL="https://www.debian.org/" +SUPPORT_URL="https://www.debian.org/support" +BUG_REPORT_URL="https://bugs.debian.org/" +``` + +If a file is located in the EDF search path, the argument to the command line option can be just the environment name, that is the name of the file without the .toml extension, for example: + +```bash +[][@-ln001 ~]$ srun --environment=debian cat /etc/os-release +PRETTY_NAME="Debian GNU/Linux 12 (bookworm)" +NAME="Debian GNU/Linux" +VERSION_ID="12" +VERSION="12 (bookworm)" +VERSION_CODENAME=bookworm +ID=debian +HOME_URL="https://www.debian.org/" +SUPPORT_URL="https://www.debian.org/support" +BUG_REPORT_URL="https://bugs.debian.org/" +``` + +### Use from batch scripts + +In principle, the --environment option can also be used within batch scripts as an #SBATCH option. +It is important to note that in such a case, all the contents of the script are executed within the containerized environment: the CE toolset gives access to the Slurm workload manager within containers via the Slurm hook, see section Container Hooks (controlled by the ENROOT_SLURM_HOOK environment variable and activated by default on most vClusters). Only with it, calls to Slurm commands (for example srun or scontrol) within the batch script will work. + + +For the time being, if the script requires to invoke Slurm commands, the recommended approach is to use --environment as part of the commands, for example, when launching job steps: + +``` +[][@-ln001 ~]$ cat example.sbatch +#!/bin/bash -l +#SBATCH --job-name=edf-example +#SBATCH --time=0:01:00 +#SBATCH --nodes=2 +#SBATCH --ntasks-per-node=1 +#SBATCH --partition= +#SBATCH --output=slurm-%x.out +  +# Run job step +srun --environment=debian cat /etc/os-release +``` + +### The EDF search path + +By default, the EDFs for each user are looked up in $HOME/.edf. The search path for EDFs can be controlled through the EDF_PATH environment variable. EDF_PATH must be a colon-separated list of absolute paths to directories where the CE looks for TOML files, similar to the PATH and LD_LIBRARY_PATH variables. If a file is located in the search path, its name can be used in --environment options without the .toml extension, for example: + +```bash +[][@-ln001 ~]$ ls -l ~/.edf +total 8 +-rw-r--r-- 1 csstaff 27 Sep 6 15:19 debian.toml + +[][@-ln001 ~]$ ls -l ~/example-project/ +total 4 +-rw-r-----+ 1 csstaff 28 Oct 26 17:44 fedora-env.toml + +[][@-ln001 ~]$ export EDF_PATH=$HOME/example-project/ + +[][@-ln001 ~]$ srun --environment=fedora-env cat /etc/os-release +NAME="Fedora Linux" +VERSION="40 (Container Image)" +ID=fedora +VERSION_ID=40 +VERSION_CODENAME="" +PLATFORM_ID="platform:f40" +PRETTY_NAME="Fedora Linux 40 (Container Image)" +[...] +``` + +## Image Management + +### Image cache + +> **NOTE**: The image caching functionality is only available on the Bristen vCluster as technical preview. + +By default, images defined in the EDF as remote registry references (e.g. a Docker reference) are automatically pulled and locally cached. A cached image would be preferred to pulling the image again in later usage.  + +An image cache is automatically created at .edf_imagestore in the user's scratch folder (i.e., $SCRATCH/.edf_imagestore), under which cached images are stored in a corresponding CPU architecture subfolder (e.g., x86 and aarch64 ). Users should regularly remove unused cache images to limit the cache size. + +Should users want to re-pull a cached image, they have to remove the corresponding image in the cache. + +To choose an alternative image store path (e.g., to use a directory owned by a group and not to an individual user), users can specify an image cache path explicitly by defining the environment variable EDF_IMAGESTORE. EDF_IMAGESTORE must be an absolute path to an existing folder. + +If the CE cannot create a directory for the image cache, it operates in cache-free mode, meaning that it pulls an ephemeral image before every container launch and discards it upon termination. + +Pulling images manually + +To work with images stored from the NGC Catalog, please refer also to the next section "Using images from third party registries and private repositories". + +To bypass any caching behavior, users can manually pull an image and directly plug it into their EDF. To do so, users may execute enroot import docker://[REGISTRY#]IMAGE[:TAG] to pull container images from OCI registries to the current directory. + +For example, the command below pulls an nvidia/cuda:11.8.0-cudnn8-devel-ubuntu22.04 image. + +enroot import docker://nvidia/cuda:11.8.0-cudnn8-devel-ubuntu22.04 +[][@-ln001 ]$ srun enroot import docker://nvidia/cuda:11.8.0-cudnn8-devel-ubuntu22.04 +[INFO] Querying registry for permission grant +[INFO] Authenticating with user: +[INFO] Authentication succeeded +[INFO] Fetching image manifest list +[INFO] Fetching image manifest +[INFO] Downloading 13 missing layers... +[INFO] Extracting image layers... +[INFO] Converting whiteouts... +[INFO] Creating squashfs filesystem... +Parallel mksquashfs: Using 64 processors +Creating 4.0 filesystem on /scratch/aistor//nvidia+cuda+11.8.0-cudnn8-devel-ubuntu22.04.sqsh, block size 131072. + +Exportable Squashfs 4.0 filesystem, zstd compressed, data block size 131072 + uncompressed data, compressed metadata, compressed fragments, + compressed xattrs, compressed ids + duplicates are removed +Filesystem size 9492185.87 Kbytes (9269.71 Mbytes) + 98.93% of uncompressed filesystem size (9594893.12 Kbytes) +Inode table size 128688 bytes (125.67 Kbytes) + 17.47% of uncompressed inode table size (736832 bytes) +Directory table size 132328 bytes (129.23 Kbytes) + 46.42% of uncompressed directory table size (285091 bytes) +Number of duplicate files found 1069 +Number of inodes 13010 +Number of files 10610 +Number of fragments 896 +Number of symbolic links 846 +Number of device nodes 0 +Number of fifo nodes 0 +Number of socket nodes 0 +Number of directories 1554 +Number of ids (unique uids + gids) 1 +Number of uids 1 + root (0) +Number of gids 1 + root (0) + +After the import is complete, images are available in Squashfs format in the current directory and can be used in EDFs, for example: + +[][@-ln001 ]$ ls -l *.sqsh +-rw-r--r-- 1 csstaff 9720037376 Sep 11 14:46 nvidia+cuda+11.8.0-cudnn8-devel-ubuntu22.04.sqsh + +[][@-ln001 ]$ realpath nvidia+cuda+11.8.0-cudnn8-devel-ubuntu22.04.sqsh  /capstor/scratch/cscs//nvidia+cuda+11.8.0-cudnn8-devel-ubuntu22.04.sqsh + +[][@-ln001 ]$ cat $HOME/.edf/cudnn8.toml +image = "/capstor/scratch/cscs//nvidia+cuda+11.8.0-cudnn8-devel-ubuntu22.04.sqsh" + +It is recommended to save images in /capstor/scratch/cscs/ or its subdirectories before using them with the CE. + +Third-party and private registries + +Docker Hub is the default registry from which remote images are imported. + +To use an image from a different registry, the corresponding registry URL has to be prepended to the image reference, using a hash character (#) as a separator. For example: + +# Usage within an EDF +[][@-ln001 ]$ cat $HOME/.edf/nvhpc-23.7.toml +image = "nvcr.io#nvidia/nvhpc:23.7-runtime-cuda11.8-ubuntu22.04" + +# Usage on the command line +[][@-ln001 ]$ srun enroot import docker://nvcr.io#nvidia/nvhpc:23.7-runtime-cuda11.8-ubuntu22.04 + +To import images from private repositories, access credentials should be configured by individual users in the $HOME/.config/enroot/.credentials file, following the netrc file format. +Using the enroot import documentation page as a reference, some examples could be: + +# NVIDIA NGC catalog (both endpoints are required) +machine nvcr.io login $oauthtoken password +machine authn.nvidia.com login $oauthtoken password + +# DockerHub +machine auth.docker.io login password + +# Google Container Registry with OAuth +machine gcr.io login oauth2accesstoken password $(gcloud auth print-access-token) +# Google Container Registry with JSON +machine gcr.io login _json_key password $(jq -c '.' $GOOGLE_APPLICATION_CREDENTIALS | sed 's/ /\\u0020/g') + +# Amazon Elastic Container Registry +machine 12345.dkr.ecr.eu-west-2.amazonaws.com login AWS password $(aws ecr get-login-password --region eu-west-2) + +# Azure Container Registry with ACR refresh token +machine myregistry.azurecr.io login 00000000-0000-0000-0000-000000000000 password $(az acr login --name myregistry --expose-token --query accessToken | tr -d '"') +# Azure Container Registry with ACR admin user +machine myregistry.azurecr.io login myregistry password $(az acr credential show --name myregistry --subscription mysub --query passwords[0].value | tr -d '"') + +# Github.com Container Registry (GITHUB_TOKEN needs read:packages scope) +machine ghcr.io login password + +# GitLab Container Registry (GITLAB_TOKEN needs a scope with read access to the container registry) +# GitLab instances often use different domains for the registry and the authentication service, respectively +# Two separate credential entries are required in such cases, for example: +# Gitlab.com +machine registry.gitlab.com login password +machine gitlab.com login password + +# ETH Zurich GitLab registry +machine registry.ethz.ch login password +machine gitlab.ethz.ch login password +Annotations  + +Annotations define arbitrary metadata for containers in the form of key-value pairs. Within the EDF, annotations are designed to be similar in appearance and behavior to those defined by the OCI Runtime Specification. Annotation keys usually express a hierarchical namespace structure, with domains separated by "." (full stop) characters. + +As annotations are often used to control hooks, they have a deep nesting level. For example, to execute the SSH hook described below, the annotation com.hooks.ssh.enabled must be set to the string "true". + +EDF files support setting annotations through the annotations table. This can be done in multiple ways in TOML: for example, both of the following usages are equivalent: + +Case: nest levels in the TOML key. + +[annotations] +com.hooks.ssh.enabled = "true" + +Case: nest levels in the TOML table name. + +[annotations.com.hooks.ssh] +enabled = "true" + +To avoid mistakes, notice a few key features of TOML: + +All property assignments belong to the section immediately preceding them (the statement in square brackets), which defines the table they refer to. +Tables, on the other hand, do not automatically belong to the tables declared before them; to nest tables, their name has to list their parents using the dot notations (so the previous example defines the table ssh inside hooks, which in turn is inside com, which is inside annotations). +An assignment can implicitly define subtables if the key you assign is a dotted list. As a reference, see the examples made earlier in this section, where assigning a string to the com.hooks.ssh.enabled attribute within the [annotations] table is exactly equivalent to assigning to the enabled attribute within the [annotations.com.hooks.ssh] subtable. +Attributes can be added to a table only in one place in the TOML file. In other words, each table must be defined in a single square bracket section. For example, both of the following are valid: +Case 1: + +[annotations.com.hooks.ssh] +authorize_ssh_key = "/capstor/scratch/cscs//tests/edf/authorized_keys" +enabled = "true" +Case 2: + +[annotations] +com.hooks.ssh.authorize_ssh_key = "/capstor/scratch/cscs//tests/edf/authorized_keys" +com.hooks.ssh.enabled = "true" +However, the following is illegal: + +[annotations] +com.hooks.ssh.authorize_ssh_key = "/capstor/scratch/cscs//tests/edf/authorized_keys" + +[annotations.com.hooks.ssh] +enabled = "true" + is not because the ssh table is defined (gets attributes set) both in the [annotations] and in the [annotations.com.hooks.ssh] sections. See the TOML format spec for more details +Accessing native resources +NVIDIA GPUs + +The Container Engine leverages components from the NVIDIA Container Toolkit to expose NVIDIA GPU devices inside containers. +GPU device files are always mounted in containers, and the NVIDIA driver user space components are  mounted if the NVIDIA_VISIBLE_DEVICES environment variable is not empty, unset or set to "void".  NVIDIA_VISIBLE_DEVICES is already set in container images officially provided by NVIDIA to enable all GPUs available on the host system. Such images are frequently used to containerize CUDA applications, either directly or as a base for custom images, thus in many cases no action is required to access GPUs. +For example, on a cluster with 4 GH200 devices per compute node: + +[][@-ln001 ~]$ cat .edf/cuda12.5.1.toml  +image = "nvidia/cuda:12.5.1-devel-ubuntu24.04" + +[][@-ln001 ~]$ srun --environment=cuda12.5.1 nvidia-smi +Thu Oct 26 17:59:36 2023        ++------------------------------------------------------------------------------------+ +| NVIDIA-SMI 535.129.03 Driver Version: 535.129.03 CUDA Version: 12.5 | +|--------------------------------------+----------------------+----------------------+ +| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC | +| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. | +| | | MIG M. | +|======================================+======================+======================| +| 0 GH200 120GB On | 00000009:01:00.0 Off | 0 | +| N/A 24C P0 89W / 900W | 37MiB / 97871MiB | 0% E. Process | +| | | Disabled | ++--------------------------------------+----------------------+----------------------+ +| 1 GH200 120GB On | 00000019:01:00.0 Off | 0 | +| N/A 24C P0 87W / 900W | 37MiB / 97871MiB | 0% E. Process | +| | | Disabled | ++--------------------------------------+----------------------+----------------------+ +| 2 GH200 120GB On | 00000029:01:00.0 Off | 0 | +| N/A 24C P0 83W / 900W | 37MiB / 97871MiB | 0% E. Process | +| | | Disabled | ++--------------------------------------+----------------------+----------------------+ +| 3 GH200 120GB On | 00000039:01:00.0 Off | 0 | +| N/A 24C P0 85W / 900W | 37MiB / 97871MiB | 0% E. Process | +| | | Disabled | ++--------------------------------------+----------------------+----------------------+ + ++------------------------------------------------------------------------------------+ +| Processes: | +| GPU GI CI PID Type Process name GPU Memory | +| ID ID Usage | +|====================================================================================| +| No running processes found | ++------------------------------------------------------------------------------------+ + +It is possible to use environment variables to control which capabilities of the NVIDIA driver are enabled inside containers. +Additionally, the NVIDIA Container Toolkit can enforce specific constraints for the container, for example, on versions of the CUDA runtime or driver, or on the architecture of the GPUs. +For the full details about using these features, please refer to the official documentation: Driver Capabilities, Constraints. + +HPE Slingshot interconnect  + +The Container Engine provides a hook to allow containers relying on libfabric to leverage the HPE Slingshot 11 high-speed interconnect. This component is commonly referred to as the "CXI hook", taking its name from the CXI libfabric provider required to interface with Slingshot 11. +The hook leverages bind-mounting the custom host libfabric library into the container (in addition to all the required dependency libraries and devices as well). +If a libfabric library is already present in the container filesystem (for example, it's provided by the image), it is replaced with its host counterpart, otherwise the host libfabric is just added to the container. + +Due to the nature of Slingshot and the mechanism implemented by the CXI hook, container applications need to use a communication library which supports libfabric in order to benefit from usage of the hook. +Libfabric support might have to be defined at compilation time (as is the case for some MPI implementations, like MPICH and OpenMPI) or could be dynamically available at runtime (as is the case with NCCL - see also this section for more details). + +The hook is activated by setting the com.hooks.cxi.enabled annotation, which can be defined in the EDF, as shown in the following example: + +# Without the CXI hook +[][@-ln001 ~]$ cat $HOME/.edf/osu-mb.toml  +image = "quay.io#madeeks/osu-mb:6.2-mpich4.1-ubuntu22.04-arm64" + +[annotations] +com.hooks.cxi.enabled = "false" + +[][@-ln001 ~]$ srun -N2 --mpi=pmi2 --environment=osu-mb ./osu_bw +# OSU MPI Bandwidth Test v6.2 +# Size Bandwidth (MB/s) +1 0.22 +2 0.40 +4 0.90 +8 1.82 +16 3.41 +32 6.81 +64 13.18 +128 26.74 +256 11.95 +512 38.06 +1024 39.65 +2048 83.22 +4096 156.14 +8192 143.08 +16384 53.78 +32768 106.77 +65536 49.88 +131072 871.86 +262144 780.97 +524288 694.58 +1048576 831.02 +2097152 1363.30 +4194304 1279.54 + + +# With the CXI hook enabling access to the Slingshot high-speed network +[][@-ln001 ~]$ cat .edf/osu-mb-cxi.toml  +image = "quay.io#madeeks/osu-mb:6.2-mpich4.1-ubuntu22.04" + +[annotations] +com.hooks.cxi.enabled = "true" + +[][@-ln001 ~]$ srun -N2 --mpi=pmi2 --environment=osu-mb-cxi ./osu_bw +# OSU MPI Bandwidth Test v6.2 +# Size Bandwidth (MB/s) +1 1.21 +2 2.32 +4 4.85 +8 8.38 +16 19.36 +32 38.47 +64 76.28 +128 151.76 +256 301.25 +512 604.17 +1024 1145.03 +2048 2367.25 +4096 4817.16 +8192 8633.36 +16384 16971.18 +32768 18740.55 +65536 21978.65 +131072 22962.31 +262144 23436.78 +524288 23672.92 +1048576 23827.78 +2097152 23890.95 +4194304 23925.61 + +On several vClusters, the CXI hook for Slingshot connectivity is enabled implicitly by default or by other hooks. +Therefore, in many cases it is not necessary to enter the enabling annotation in the EDF. + +Container Hooks + +Container hooks let you customize container behavior to fit system-specific needs, making them especially valuable for High-Performance Computing. + +What they do: Hooks extend container runtime functionality by enabling custom actions during a container's lifecycle. +Use for HPC: HPC systems rely on specialized hardware and fine-tuned software, unlike generic containers. Hooks bridge this gap by allowing containers to access these system-specific resources or enable custom features. + +This section outlines all hooks supported in production by the Container Engine. However, specific Alps vClusters may support only a subset or use custom configurations. For details about available features in individual vClusters, consult platform documentation or contact CSCS support. + +AWS OFI NCCL Hook  + +The AWS OFI NCCL plugin is a software extension that allows the NCCL and RCCL libraries to use libfabric as a network provider and, through libfabric, to access the Slingshot high-speed interconnect. + + +The Container Engine includes a hook program to inject the AWS OFI NCCL plugin in containers; since the plugin must also be compatible with the GPU programming software stack being used, the com.hooks.aws_ofi_nccl.variant annotation is used to specify a plugin variant suitable for a given container image. +At the moment of writing, 4 plugin variants are configured: cuda11, cuda12 (to be used on NVIDIA GPU nodes), rocm5, and rocm6 (to be used on AMD GPU nodes alongside RCCL). +For example, the following EDF enables the hook and uses it to mount the plugin in a CUDA 11 image: + +image = "nvcr.io#nvidia/pytorch:22.12-py3" +mounts = ["/capstor/scratch/cscs/amadonna:/capstor/scratch/cscs/amadonna"] +entrypoint = false + +[annotations] +com.hooks.aws_ofi_nccl.enabled = "true" +com.hooks.aws_ofi_nccl.variant = "cuda11" + +The AWS OFI NCCL hook also takes care of the following aspects: + +It implicitly enables the CXI hook, therefore exposing the Slingshot interconnect to container applications. In other words, when enabling the AWS OFI NCCL hook, it's unnecessary to also enable the CXI hook separately in the EDF. +It sets environment variables to control the behavior of NCCL and the libfabric CXI provider for Slingshot. In particular, the NCCL_NET_PLUGIN variable is set to force NCCL to load the specific network plugin mounted by the hook. This is useful because certain container images (for example, those from NGC repositories) might already ship with a default NCCL plugin. Other environment variables help prevent application stalls and improve performance when using GPUDirect for RDMA communication. +SSH Hook + +The SSH hook runs a lightweight, statically-linked SSH server (a build of Dropbear) inside the container. It can be useful to add SSH connectivity to containers (for example, enabling remote debugging) without bundling an SSH server into the container image or creating ad-hoc image variants for such purposes. + +The com.hooks.ssh.authorize_ssh_key annotation allows the authorization of a custom public SSH key for remote connections. The annotation value must be the absolute path to a text file containing the public key (just the public key without any extra signature/certificate). After the container starts, it is possible to get a remote shell inside the container by connecting with SSH to the listening port. + +By default, the server started by the SSH hook listens to port 15263, but this setting can be controlled through the com.hooks.ssh.port annotation in the EDF. + +It is required to keep the container writable to use the hook. + +The following EDF file shows an example of enabling the SSH hook and authorizing a user-provided public key: + +[][@-ln001 ~]$ cat $HOME/.edf/ubuntu-ssh.toml +image = "ubuntu:latest" +writable = true + +[annotations.com.hooks.ssh] +enabled = "true" +authorize_ssh_key = "" + +Using the previous EDF, a container can be started as follows. Notice that the --pty option for the srun command is currently required in order for the hook to initialize properly: + +[][@-ln001 ~]$ srun --environment=ubuntu-ssh --pty + +While the container is running, it's possible to connect to it from a remote host using a private key matching the public one authorized in the EDF annotation. For example, in a host where such private key is the default identity file, the following command could be used: + +ssh -p 15263 + + + + +In order to establish connections through Visual Studio Code Remote - SSH extension, the scp program must be available within the container. This is required to send and establish the VS Code Server into the remote container. + +NVIDIA CUDA MPS Hook + +On several Alps vClusters, NVIDIA GPUs by default operate in "Exclusive process" mode, that is, the CUDA driver is configured to allow only one process at a time to use a given GPU. +For example, on a node with 4 GPUs, a maximum of 4 CUDA processes can run at the same time: + +[][@-ln001 ~]$ nvidia-smi -L +GPU 0: GH200 120GB (UUID: GPU-...) +GPU 1: GH200 120GB (UUID: GPU-...) +GPU 2: GH200 120GB (UUID: GPU-...) +GPU 3: GH200 120GB (UUID: GPU-...) + +# This EDF uses the CUDA vector addition sample from NVIDIA's NGC catalog +[][@-ln001 ~]$ cat $HOME/.edf/vectoradd-cuda.toml +image = "nvcr.io#nvidia/k8s/cuda-sample:vectoradd-cuda12.5.0-ubuntu22.04" + +# 4 processes run successfully +[][@-ln001 ~]$ srun -t2 -N1 -n4 --environment=vectoradd-cuda /cuda-samples/vectorAdd | grep "Test PASSED" +Test PASSED +Test PASSED +Test PASSED +Test PASSED + +# More than 4 concurrent processes result in oversubscription errors +[][@-ln001 ~]$ srun -t2 -N1 -n5 --environment=vectoradd-cuda /cuda-samples/vectorAdd | grep "Test PASSED" +Failed to allocate device vector A (error code CUDA-capable device(s) is/are busy or unavailable)! +srun: error: [...] +[...] + +In order to run multiple processes concurrently on the same GPU (one example could be running multiple MPI ranks on the same device), the NVIDIA CUDA Multi-Process Service (or MPS, for short) must be started on the compute node. + +The Container Engine provides a hook to automatically manage the setup and removal of the NVIDIA CUDA MPS components within containers. +The hook can be activated by setting the com.hooks.nvidia_cuda_mps.enabled to the string true. + +It is required to keep the container writable to be able to use the hook. + +The following is an example of using the NVIDIA CUDA MPS hook: + +[][@-ln001 ~]$ cat $HOME/.edf/vectoradd-cuda-mps.toml +image = "nvcr.io#nvidia/k8s/cuda-sample:vectoradd-cuda12.5.0-ubuntu22.04" +writable = true + +[annotations] +com.hooks.nvidia_cuda_mps.enabled = "true" + +[][@-ln001 ~]$ srun -t2 -N1 -n8 --environment=vectoradd-cuda-mps /cuda-samples/vectorAdd | grep "Test PASSED" | wc -l +8 + +When using the NVIDIA CUDA MPS hook it is not necessary to use other wrappers or scripts to manage the Multi-Process Service, as is documented for native jobs on some vClusters. + +EDF Reference + +EDF files use the TOML format. For details about the data types used by the different parameters, please refer to the TOML spec webpage. + +Parameter Description + +base_environment + +Type: ARRAY or BASIC STRING + +Default: N/A + + + +Ordered list of EDFs that this file inherits from. Parameters from listed environments are evaluated sequentially. Supports up to 10 levels of recursion. + +Parameters from the listed environments are evaluated sequentially, adding new entries or overwriting previous ones, before evaluating the parameters from the current EDF. In other words, the current EDF inherits the parameters from the EDFs listed in base_environment. When evaluating mounts or env parameters, values from downstream EDFs are appended to inherited values. +The individual EDF entries in the array follow the same search rules as the arguments of the --environment CLI option for Slurm; they can be either file paths or filenames without extension if the file is located in the EDF search path. +This parameter can be a string if there is only one base environment. + + + +Examples + +1. Single environment inheritance: + +base_environment = "common_env" + +2. Multiple environment inheritance: + +base_environment = ["common_env", "ml_pytorch_env1"] + + +image + +Type: BASIC STRING + +Default: N/A + + + +The container image to use. Can reference a remote Docker/OCI registry or a local Squashfs file as a filesystem path. + +The full format for remote references is [USER@][REGISTRY#]IMAGE[:TAG]. +[REGISTRY#] : (optional) registry URL, followed by #. Default: Docker Hub. +IMAGE : image name. +[:TAG] : (optional) image tag name, preceded by :. +The registry user can also be specified in the $HOME/.config/enroot/.credentials file. +The default registry is Docker Hub (docker.io). + + + +Examples + +1. Reference of Ubuntu image in the Docker Hub registry (default registry) + +image = "library/ubuntu:24.04" + +2. Explicit reference of Ubuntu image in the Docker Hub registry + +image = "docker.io#library/ubuntu:24.04" + +3. Reference to PyTorch image from NVIDIA Container Registry (nvcr.io) + +image = "nvcr.io#nvidia/pytorch:22.12-py3" + +5. Image from third-party quay.io registry + +image = "quay.io#madeeks/osu-mb:6.2-mpich4.1-ubuntu22.04-arm64" + +4. Reference to a manually pulled image stored in parallel FS + +image = "/path/to/image.squashfs" + + +workdir + +Type: BASIC STRING + +Default: + +Inherited from image + + + +Initial working directory when the container starts. + + + + +Examples + +1. Workdir pointing to a user defined project path  + +workdir = "/home/user/projects" + +2. Workdir pointing to the /tmp directory + +workdir = "/tmp" + + +entrypoint + +Type: BOOL + +Default: true + + + +If true, run the entrypoint from the container image + + + + +Examples + +entrypoint = false + + + + +writable + +Type: BOOL + +Default: false + + + +If false, the container filesystem is read-only + + + + +Examples + +writable = true + + + + +mounts + +Type: ARRAY + +Default: N/A + + + +List of bind mounts in the format SOURCE:DESTINATION[:FLAGS]. Flags are optional and can include ro, private, etc. + +Mount flags are separated with a plus symbol, for example: ro+private. +Optional flags from docker format or OCI (need reference) + + + + + + + +Examples + +Literal fixed mount map +mounts = ["/capstor/scratch/cscs/amadonna:/capstor/scratch/cscs/amadonna"] + +2. Mapping path with env variable expansion + +mounts = ["/capstor/scratch/cscs/${USER}:/capstor/scratch/cscs/${USER}"] + +3. Mounting the scratch filesystem using a host environment variable + +mounts = ["${SCRATCH}:/scratch"] + + + + +env + +Type: TABLE + +Default: + +Inherited from + +host and + +image + + + +Environment variables to set in the container. Null-string values will unset the variable. + +By default, containers inherit environment variables from the container image and the host environment, with variables from the image taking precedence. +The env table can be used to further customize the container environment by setting, modifying, or unsetting variables. +Values of the table entries must be strings. If an entry has a null value, the variable corresponding to the entry key is unset in the container. + + + +Examples + +1. Basic env block + +[env] +MY_RUN = "production", +DEBUG = "false" +2. Use of environment variable expansion +[env] +MY_NODE = "${VAR_FROM_HOST}", +PATH = "${PATH}:/custom/bin", +DEBUG = "true" + + + + + + + + +annotations + +Type: TABLE + +Default: N/A + + + +OCI-like annotations for the container. + +For more details, refer to the Annotations section. + +  + + + + +Examples + +1. Disabling the CXI hook + +[annotations] +com.hooks.cxi.enabled = "false" + +2. Control of SSH hook parameters via annotation and variable expansion + +[annotations.com.hooks.ssh] +authorize_ssh_key = "/capstor/scratch/cscs/${USER}/tests/edf/authorized_keys" +enabled = "true" + +3. Alternative example for usage of annotation with fixed path + +[annotations] +com.hooks.ssh.authorize_ssh_key = "/path/to/authorized_keys" +com.hooks.ssh.enabled = "true" + + + + + + +Environment variable expansion and relative paths expansion are only available on the Bristen vCluster as technical preview. + +Environment Variable Expansion + +Environment variable expansion allows for dynamic substitution of environment variable values within the EDF (Environment Definition File). This capability applies across all configuration parameters in the EDF, providing flexibility in defining container environments. + +Syntax. Use ${VAR} to reference an environment variable VAR. The variable's value is resolved from the combined environment, which includes variables defined in the host and the container image, the later taking precedence. +Scope. Variable expansion is supported across all EDF parameters. This includes EDF’s parameters like mounts, workdir, image, etc. For example, ${SCRATCH} can be used in mounts to reference a directory path. +Undefined Variables. Referencing an undefined variable results in an error. To safely handle undefined variables, you can use the syntax ${VAR:-}, which evaluates to an empty string if VAR is undefined. +Preventing Expansion. To prevent expansion, use double dollar signs $$. For example, $$${VAR} will render as the literal string ${VAR}. +Limitations: +Variables defined within the [env] EDF table cannot reference other entries from [env] tables in the same or other EDF files (e.g. the ones entered as base environments) . Therefore, only environment variables from the host or image can be referenced. +Environment Variable Resolution Order. The environment variables are resolved based on the following order: +TOML env: Variable values as defined in EDF’s env. +Container Image: Variables defined in the container image's environment take precedence. +Host Environment: Environment variables defined in the host system. +Relative paths expansion + +Relative filesystem paths can be used within EDF parameters, and will be expanded by the CE at runtime. The paths are interpreted as relative to the working directory of the process calling the CE, not to the location of the EDF file. + +Known Issues +Compatibility with Alpine Linux + +Alpine Linux is incompatible with some hooks, causing errors when used with Slurm. For example, + +[][@-ln001 ~]$ cat alpine.toml +image = "alpine:3.19" +[][@-ln001 ~]$ srun -lN1 --environment=alpine.toml echo "abc" +0: slurmstepd: error: pyxis: container start failed with error code: 1 +0: slurmstepd: error: pyxis: printing enroot log file: +0: slurmstepd: error: pyxis: [ERROR] Failed to refresh the dynamic linker cache +0: slurmstepd: error: pyxis: [ERROR] /etc/enroot/hooks.d/87-slurm.sh exited with return code 1 +0: slurmstepd: error: pyxis: couldn't start container +0: slurmstepd: error: spank: required plugin spank_pyxis.so: task_init() failed with rc=-1 +0: slurmstepd: error: Failed to invoke spank plugin stack + +This is because some hooks (e.g., Slurm and CXI hooks) leverage ldconfig (from Glibc) when they bind-mount host libraries inside containers; since Alpine Linux provides an alternative ldconfig (from Musl Libc), it does not work as intended by hooks. As a workaround, users may disable problematic hooks. For example, + +[][@-ln001 ~]$ cat alpine_workaround.toml +image = "alpine:3.19" +[annotations] +com.hooks.slurm.enabled = "false" +com.hooks.cxi.enabled = "false" +[][@-ln001 ~]$ srun -lN1 --environment=alpine_workaround.toml echo "abc" +abc + +Notice the section [annotations] disabling Slurm and CXI hooks. + +Further reading +Containerized use case examples +Building container images on Alps From 38e8e87973142871f518fcadd9e452f5daa2ba79 Mon Sep 17 00:00:00 2001 From: gwangmu <43980405+gwangmu@users.noreply.github.com> Date: Tue, 25 Feb 2025 11:22:34 +0100 Subject: [PATCH 02/34] Update container-engine.md --- docs/tools/container-engine.md | 44 ++++++++++++++++++++++++---------- 1 file changed, 31 insertions(+), 13 deletions(-) diff --git a/docs/tools/container-engine.md b/docs/tools/container-engine.md index bbeb252e..c09099e4 100644 --- a/docs/tools/container-engine.md +++ b/docs/tools/container-engine.md @@ -25,7 +25,7 @@ Let's set up a containerized Ubuntu 24.04 environment using a scratch folder as ### Example EDF -``` +```bash image = "library/ubuntu:24.04" mounts = ["/capstor/scratch/cscs/${USER}:/capstor/scratch/cscs/${USER}"] workdir = "/capstor/scratch/cscs/${USER}" @@ -39,7 +39,7 @@ Save this file as ubuntu.toml file in $HOME/.edf directory (which is the defaul Use Slurm in the cluster login node to start the Ubuntu environment that was just defined as follows: -``` +```bash $ srun --environment=ubuntu --pty bash ``` @@ -180,7 +180,7 @@ PRETTY_NAME="Fedora Linux 40 (Container Image)" ### Image cache -> **NOTE**: The image caching functionality is only available on the Bristen vCluster as technical preview. +> **INFO**: The image caching functionality is only available on the Bristen vCluster as technical preview. By default, images defined in the EDF as remote registry references (e.g. a Docker reference) are automatically pulled and locally cached. A cached image would be preferred to pulling the image again in later usage.  @@ -190,9 +190,9 @@ Should users want to re-pull a cached image, they have to remove the correspondi To choose an alternative image store path (e.g., to use a directory owned by a group and not to an individual user), users can specify an image cache path explicitly by defining the environment variable EDF_IMAGESTORE. EDF_IMAGESTORE must be an absolute path to an existing folder. -If the CE cannot create a directory for the image cache, it operates in cache-free mode, meaning that it pulls an ephemeral image before every container launch and discards it upon termination. +> **NOTE**: If the CE cannot create a directory for the image cache, it operates in cache-free mode, meaning that it pulls an ephemeral image before every container launch and discards it upon termination. -Pulling images manually +### Pulling images manually To work with images stored from the NGC Catalog, please refer also to the next section "Using images from third party registries and private repositories". @@ -200,7 +200,14 @@ To bypass any caching behavior, users can manually pull an image and directly pl For example, the command below pulls an nvidia/cuda:11.8.0-cudnn8-devel-ubuntu22.04 image. -enroot import docker://nvidia/cuda:11.8.0-cudnn8-devel-ubuntu22.04 +```bash +$ enroot import docker://nvidia/cuda:11.8.0-cudnn8-devel-ubuntu22.04 +``` + +
+Image import w/ full output + +```bash [][@-ln001 ]$ srun enroot import docker://nvidia/cuda:11.8.0-cudnn8-devel-ubuntu22.04 [INFO] Querying registry for permission grant [INFO] Authenticating with user: @@ -238,9 +245,12 @@ Number of uids 1 root (0) Number of gids 1 root (0) +``` +
After the import is complete, images are available in Squashfs format in the current directory and can be used in EDFs, for example: +```bash [][@-ln001 ]$ ls -l *.sqsh -rw-r--r-- 1 csstaff 9720037376 Sep 11 14:46 nvidia+cuda+11.8.0-cudnn8-devel-ubuntu22.04.sqsh @@ -248,25 +258,29 @@ After the import is complete, images are available in Squashfs format in the cur [][@-ln001 ]$ cat $HOME/.edf/cudnn8.toml image = "/capstor/scratch/cscs//nvidia+cuda+11.8.0-cudnn8-devel-ubuntu22.04.sqsh" +``` -It is recommended to save images in /capstor/scratch/cscs/ or its subdirectories before using them with the CE. +> **NOTE**: It is recommended to save images in /capstor/scratch/cscs/ or its subdirectories before using them with the CE. -Third-party and private registries +### Third-party and private registries Docker Hub is the default registry from which remote images are imported. To use an image from a different registry, the corresponding registry URL has to be prepended to the image reference, using a hash character (#) as a separator. For example: +```bash # Usage within an EDF [][@-ln001 ]$ cat $HOME/.edf/nvhpc-23.7.toml image = "nvcr.io#nvidia/nvhpc:23.7-runtime-cuda11.8-ubuntu22.04" # Usage on the command line [][@-ln001 ]$ srun enroot import docker://nvcr.io#nvidia/nvhpc:23.7-runtime-cuda11.8-ubuntu22.04 +``` To import images from private repositories, access credentials should be configured by individual users in the $HOME/.config/enroot/.credentials file, following the netrc file format. Using the enroot import documentation page as a reference, some examples could be: +```bash # NVIDIA NGC catalog (both endpoints are required) machine nvcr.io login $oauthtoken password machine authn.nvidia.com login $oauthtoken password @@ -300,7 +314,9 @@ machine gitlab.com login password # ETH Zurich GitLab registry machine registry.ethz.ch login password machine gitlab.ethz.ch login password -Annotations  +``` + +## Annotations  Annotations define arbitrary metadata for containers in the form of key-value pairs. Within the EDF, annotations are designed to be similar in appearance and behavior to those defined by the OCI Runtime Specification. Annotation keys usually express a hierarchical namespace structure, with domains separated by "." (full stop) characters. @@ -308,15 +324,17 @@ As annotations are often used to control hooks, they have a deep nesting level. EDF files support setting annotations through the annotations table. This can be done in multiple ways in TOML: for example, both of the following usages are equivalent: -Case: nest levels in the TOML key. - + * Case: nest levels in the TOML key. +```bash [annotations] com.hooks.ssh.enabled = "true" +``` -Case: nest levels in the TOML table name. - + * Case: nest levels in the TOML table name. +```bash [annotations.com.hooks.ssh] enabled = "true" +``` To avoid mistakes, notice a few key features of TOML: From 07d5548d568d2569ef2c5cfbd3b01a375ab5f076 Mon Sep 17 00:00:00 2001 From: gwangmu <43980405+gwangmu@users.noreply.github.com> Date: Tue, 25 Feb 2025 11:37:47 +0100 Subject: [PATCH 03/34] Update container-engine.md --- docs/tools/container-engine.md | 126 +++++++++++++++++++-------------- 1 file changed, 72 insertions(+), 54 deletions(-) diff --git a/docs/tools/container-engine.md b/docs/tools/container-engine.md index c09099e4..f5da13f8 100644 --- a/docs/tools/container-engine.md +++ b/docs/tools/container-engine.md @@ -338,35 +338,40 @@ enabled = "true" To avoid mistakes, notice a few key features of TOML: -All property assignments belong to the section immediately preceding them (the statement in square brackets), which defines the table they refer to. -Tables, on the other hand, do not automatically belong to the tables declared before them; to nest tables, their name has to list their parents using the dot notations (so the previous example defines the table ssh inside hooks, which in turn is inside com, which is inside annotations). -An assignment can implicitly define subtables if the key you assign is a dotted list. As a reference, see the examples made earlier in this section, where assigning a string to the com.hooks.ssh.enabled attribute within the [annotations] table is exactly equivalent to assigning to the enabled attribute within the [annotations.com.hooks.ssh] subtable. -Attributes can be added to a table only in one place in the TOML file. In other words, each table must be defined in a single square bracket section. For example, both of the following are valid: -Case 1: - + * All property assignments belong to the section immediately preceding them (the statement in square brackets), which defines the table they refer to. + * Tables, on the other hand, do not automatically belong to the tables declared before them; to nest tables, their name has to list their parents using the dot notations (so the previous example defines the table ssh inside hooks, which in turn is inside com, which is inside annotations). + * An assignment can implicitly define subtables if the key you assign is a dotted list. As a reference, see the examples made earlier in this section, where assigning a string to the com.hooks.ssh.enabled attribute within the [annotations] table is exactly equivalent to assigning to the enabled attribute within the [annotations.com.hooks.ssh] subtable. + * Attributes can be added to a table only in one place in the TOML file. In other words, each table must be defined in a single square bracket section. For example, Case 3 in the example below is invalid because the ssh table is defined (gets attributes set) both in the [annotations] and in the [annotations.com.hooks.ssh] sections. See the TOML format spec for more details. + * Case 1 (valid): +```bash [annotations.com.hooks.ssh] authorize_ssh_key = "/capstor/scratch/cscs//tests/edf/authorized_keys" enabled = "true" -Case 2: - +``` + * Case 2 (valid): +```bash [annotations] com.hooks.ssh.authorize_ssh_key = "/capstor/scratch/cscs//tests/edf/authorized_keys" com.hooks.ssh.enabled = "true" -However, the following is illegal: - +``` + * Case 3 (**invalid**): +```bash [annotations] com.hooks.ssh.authorize_ssh_key = "/capstor/scratch/cscs//tests/edf/authorized_keys" [annotations.com.hooks.ssh] enabled = "true" - is not because the ssh table is defined (gets attributes set) both in the [annotations] and in the [annotations.com.hooks.ssh] sections. See the TOML format spec for more details -Accessing native resources -NVIDIA GPUs +``` + +## Accessing native resources + +### NVIDIA GPUs The Container Engine leverages components from the NVIDIA Container Toolkit to expose NVIDIA GPU devices inside containers. GPU device files are always mounted in containers, and the NVIDIA driver user space components are  mounted if the NVIDIA_VISIBLE_DEVICES environment variable is not empty, unset or set to "void".  NVIDIA_VISIBLE_DEVICES is already set in container images officially provided by NVIDIA to enable all GPUs available on the host system. Such images are frequently used to containerize CUDA applications, either directly or as a base for custom images, thus in many cases no action is required to access GPUs. For example, on a cluster with 4 GH200 devices per compute node: +```bash [][@-ln001 ~]$ cat .edf/cuda12.5.1.toml  image = "nvidia/cuda:12.5.1-devel-ubuntu24.04" @@ -403,22 +408,24 @@ Thu Oct 26 17:59:36 2023        |====================================================================================| | No running processes found | +------------------------------------------------------------------------------------+ +``` It is possible to use environment variables to control which capabilities of the NVIDIA driver are enabled inside containers. Additionally, the NVIDIA Container Toolkit can enforce specific constraints for the container, for example, on versions of the CUDA runtime or driver, or on the architecture of the GPUs. For the full details about using these features, please refer to the official documentation: Driver Capabilities, Constraints. -HPE Slingshot interconnect  +### HPE Slingshot interconnect  The Container Engine provides a hook to allow containers relying on libfabric to leverage the HPE Slingshot 11 high-speed interconnect. This component is commonly referred to as the "CXI hook", taking its name from the CXI libfabric provider required to interface with Slingshot 11. The hook leverages bind-mounting the custom host libfabric library into the container (in addition to all the required dependency libraries and devices as well). If a libfabric library is already present in the container filesystem (for example, it's provided by the image), it is replaced with its host counterpart, otherwise the host libfabric is just added to the container. -Due to the nature of Slingshot and the mechanism implemented by the CXI hook, container applications need to use a communication library which supports libfabric in order to benefit from usage of the hook. -Libfabric support might have to be defined at compilation time (as is the case for some MPI implementations, like MPICH and OpenMPI) or could be dynamically available at runtime (as is the case with NCCL - see also this section for more details). +> **NOTE**: Due to the nature of Slingshot and the mechanism implemented by the CXI hook, container applications need to use a communication library which supports libfabric in order to benefit from usage of the hook. +> Libfabric support might have to be defined at compilation time (as is the case for some MPI implementations, like MPICH and OpenMPI) or could be dynamically available at runtime (as is the case with NCCL - see also this section for more details). The hook is activated by setting the com.hooks.cxi.enabled annotation, which can be defined in the EDF, as shown in the following example: +```bash # Without the CXI hook [][@-ln001 ~]$ cat $HOME/.edf/osu-mb.toml  image = "quay.io#madeeks/osu-mb:6.2-mpich4.1-ubuntu22.04-arm64" @@ -487,28 +494,28 @@ com.hooks.cxi.enabled = "true" 1048576 23827.78 2097152 23890.95 4194304 23925.61 +``` -On several vClusters, the CXI hook for Slingshot connectivity is enabled implicitly by default or by other hooks. -Therefore, in many cases it is not necessary to enter the enabling annotation in the EDF. +> **TIP**: On several vClusters, the CXI hook for Slingshot connectivity is enabled implicitly by default or by other hooks. Therefore, entering the enabling annotation in the EDF is unnecessary in many cases. -Container Hooks +## Container Hooks Container hooks let you customize container behavior to fit system-specific needs, making them especially valuable for High-Performance Computing. -What they do: Hooks extend container runtime functionality by enabling custom actions during a container's lifecycle. -Use for HPC: HPC systems rely on specialized hardware and fine-tuned software, unlike generic containers. Hooks bridge this gap by allowing containers to access these system-specific resources or enable custom features. + * *What they do*: Hooks extend container runtime functionality by enabling custom actions during a container's lifecycle. + * *Use for HPC*: HPC systems rely on specialized hardware and fine-tuned software, unlike generic containers. Hooks bridge this gap by allowing containers to access these system-specific resources or enable custom features. -This section outlines all hooks supported in production by the Container Engine. However, specific Alps vClusters may support only a subset or use custom configurations. For details about available features in individual vClusters, consult platform documentation or contact CSCS support. +> **INFO**: This section outlines all hooks supported in production by the Container Engine. However, specific Alps vClusters may support only a subset or use custom configurations. For details about available features in individual vClusters, consult platform documentation or contact CSCS support. -AWS OFI NCCL Hook  +### AWS OFI NCCL Hook  The AWS OFI NCCL plugin is a software extension that allows the NCCL and RCCL libraries to use libfabric as a network provider and, through libfabric, to access the Slingshot high-speed interconnect. - The Container Engine includes a hook program to inject the AWS OFI NCCL plugin in containers; since the plugin must also be compatible with the GPU programming software stack being used, the com.hooks.aws_ofi_nccl.variant annotation is used to specify a plugin variant suitable for a given container image. At the moment of writing, 4 plugin variants are configured: cuda11, cuda12 (to be used on NVIDIA GPU nodes), rocm5, and rocm6 (to be used on AMD GPU nodes alongside RCCL). For example, the following EDF enables the hook and uses it to mount the plugin in a CUDA 11 image: +```bash image = "nvcr.io#nvidia/pytorch:22.12-py3" mounts = ["/capstor/scratch/cscs/amadonna:/capstor/scratch/cscs/amadonna"] entrypoint = false @@ -516,12 +523,14 @@ entrypoint = false [annotations] com.hooks.aws_ofi_nccl.enabled = "true" com.hooks.aws_ofi_nccl.variant = "cuda11" +``` The AWS OFI NCCL hook also takes care of the following aspects: -It implicitly enables the CXI hook, therefore exposing the Slingshot interconnect to container applications. In other words, when enabling the AWS OFI NCCL hook, it's unnecessary to also enable the CXI hook separately in the EDF. -It sets environment variables to control the behavior of NCCL and the libfabric CXI provider for Slingshot. In particular, the NCCL_NET_PLUGIN variable is set to force NCCL to load the specific network plugin mounted by the hook. This is useful because certain container images (for example, those from NGC repositories) might already ship with a default NCCL plugin. Other environment variables help prevent application stalls and improve performance when using GPUDirect for RDMA communication. -SSH Hook + * It implicitly enables the CXI hook, therefore exposing the Slingshot interconnect to container applications. In other words, when enabling the AWS OFI NCCL hook, it's unnecessary to also enable the CXI hook separately in the EDF. + * It sets environment variables to control the behavior of NCCL and the libfabric CXI provider for Slingshot. In particular, the NCCL_NET_PLUGIN variable is set to force NCCL to load the specific network plugin mounted by the hook. This is useful because certain container images (for example, those from NGC repositories) might already ship with a default NCCL plugin. Other environment variables help prevent application stalls and improve performance when using GPUDirect for RDMA communication. + +### SSH Hook The SSH hook runs a lightweight, statically-linked SSH server (a build of Dropbear) inside the container. It can be useful to add SSH connectivity to containers (for example, enabling remote debugging) without bundling an SSH server into the container image or creating ad-hoc image variants for such purposes. @@ -529,10 +538,11 @@ The com.hooks.ssh.authorize_ssh_key annotation allows the authorization of a cu By default, the server started by the SSH hook listens to port 15263, but this setting can be controlled through the com.hooks.ssh.port annotation in the EDF. -It is required to keep the container writable to use the hook. +> **NOTE**: To use the SSH hook, it is **required** to keep the container **writable**. The following EDF file shows an example of enabling the SSH hook and authorizing a user-provided public key: +```bash [][@-ln001 ~]$ cat $HOME/.edf/ubuntu-ssh.toml image = "ubuntu:latest" writable = true @@ -540,25 +550,28 @@ writable = true [annotations.com.hooks.ssh] enabled = "true" authorize_ssh_key = "" +``` Using the previous EDF, a container can be started as follows. Notice that the --pty option for the srun command is currently required in order for the hook to initialize properly: +```bash [][@-ln001 ~]$ srun --environment=ubuntu-ssh --pty +``` While the container is running, it's possible to connect to it from a remote host using a private key matching the public one authorized in the EDF annotation. For example, in a host where such private key is the default identity file, the following command could be used: +```bash ssh -p 15263 +``` +> **INFO**: In order to establish connections through Visual Studio Code Remote - SSH extension, the scp program must be available within the container. This is required to send and establish the VS Code Server into the remote container. - - -In order to establish connections through Visual Studio Code Remote - SSH extension, the scp program must be available within the container. This is required to send and establish the VS Code Server into the remote container. - -NVIDIA CUDA MPS Hook +### NVIDIA CUDA MPS Hook On several Alps vClusters, NVIDIA GPUs by default operate in "Exclusive process" mode, that is, the CUDA driver is configured to allow only one process at a time to use a given GPU. For example, on a node with 4 GPUs, a maximum of 4 CUDA processes can run at the same time: +```bash [][@-ln001 ~]$ nvidia-smi -L GPU 0: GH200 120GB (UUID: GPU-...) GPU 1: GH200 120GB (UUID: GPU-...) @@ -581,16 +594,18 @@ Test PASSED Failed to allocate device vector A (error code CUDA-capable device(s) is/are busy or unavailable)! srun: error: [...] [...] +``` In order to run multiple processes concurrently on the same GPU (one example could be running multiple MPI ranks on the same device), the NVIDIA CUDA Multi-Process Service (or MPS, for short) must be started on the compute node. The Container Engine provides a hook to automatically manage the setup and removal of the NVIDIA CUDA MPS components within containers. The hook can be activated by setting the com.hooks.nvidia_cuda_mps.enabled to the string true. -It is required to keep the container writable to be able to use the hook. +> **NOTE**: To use the CUDA MPS hook, it is **required** to keep the container **writable**. The following is an example of using the NVIDIA CUDA MPS hook: +```bash [][@-ln001 ~]$ cat $HOME/.edf/vectoradd-cuda-mps.toml image = "nvcr.io#nvidia/k8s/cuda-sample:vectoradd-cuda12.5.0-ubuntu22.04" writable = true @@ -600,10 +615,11 @@ com.hooks.nvidia_cuda_mps.enabled = "true" [][@-ln001 ~]$ srun -t2 -N1 -n8 --environment=vectoradd-cuda-mps /cuda-samples/vectorAdd | grep "Test PASSED" | wc -l 8 +``` -When using the NVIDIA CUDA MPS hook it is not necessary to use other wrappers or scripts to manage the Multi-Process Service, as is documented for native jobs on some vClusters. +> **INFO**: When using the NVIDIA CUDA MPS hook it is not necessary to use other wrappers or scripts to manage the Multi-Process Service, as is documented for native jobs on some vClusters. -EDF Reference +## EDF Reference EDF files use the TOML format. For details about the data types used by the different parameters, please refer to the TOML spec webpage. @@ -864,29 +880,32 @@ com.hooks.ssh.enabled = "true" Environment variable expansion and relative paths expansion are only available on the Bristen vCluster as technical preview. -Environment Variable Expansion +### Environment Variable Expansion Environment variable expansion allows for dynamic substitution of environment variable values within the EDF (Environment Definition File). This capability applies across all configuration parameters in the EDF, providing flexibility in defining container environments. -Syntax. Use ${VAR} to reference an environment variable VAR. The variable's value is resolved from the combined environment, which includes variables defined in the host and the container image, the later taking precedence. -Scope. Variable expansion is supported across all EDF parameters. This includes EDF’s parameters like mounts, workdir, image, etc. For example, ${SCRATCH} can be used in mounts to reference a directory path. -Undefined Variables. Referencing an undefined variable results in an error. To safely handle undefined variables, you can use the syntax ${VAR:-}, which evaluates to an empty string if VAR is undefined. -Preventing Expansion. To prevent expansion, use double dollar signs $$. For example, $$${VAR} will render as the literal string ${VAR}. -Limitations: -Variables defined within the [env] EDF table cannot reference other entries from [env] tables in the same or other EDF files (e.g. the ones entered as base environments) . Therefore, only environment variables from the host or image can be referenced. -Environment Variable Resolution Order. The environment variables are resolved based on the following order: -TOML env: Variable values as defined in EDF’s env. -Container Image: Variables defined in the container image's environment take precedence. -Host Environment: Environment variables defined in the host system. -Relative paths expansion + * *Syntax*. Use ${VAR} to reference an environment variable VAR. The variable's value is resolved from the combined environment, which includes variables defined in the host and the container image, the later taking precedence. + * *Scope*. Variable expansion is supported across all EDF parameters. This includes EDF’s parameters like mounts, workdir, image, etc. For example, ${SCRATCH} can be used in mounts to reference a directory path. + * *Undefined Variables*. Referencing an undefined variable results in an error. To safely handle undefined variables, you can use the syntax ${VAR:-}, which evaluates to an empty string if VAR is undefined. + * *Preventing Expansion*. To prevent expansion, use double dollar signs $$. For example, $$${VAR} will render as the literal string ${VAR}. + * *Limitations* + * Variables defined within the [env] EDF table cannot reference other entries from [env] tables in the same or other EDF files (e.g. the ones entered as base environments) . Therefore, only environment variables from the host or image can be referenced. + * *Environment Variable Resolution Order*. The environment variables are resolved based on the following order: + 1. TOML env: Variable values as defined in EDF’s env. + 2. Container Image: Variables defined in the container image's environment take precedence. + 3. Host Environment: Environment variables defined in the host system. + +### Relative paths expansion Relative filesystem paths can be used within EDF parameters, and will be expanded by the CE at runtime. The paths are interpreted as relative to the working directory of the process calling the CE, not to the location of the EDF file. -Known Issues -Compatibility with Alpine Linux +## Known Issues + +### Compatibility with Alpine Linux Alpine Linux is incompatible with some hooks, causing errors when used with Slurm. For example, +```bash [][@-ln001 ~]$ cat alpine.toml image = "alpine:3.19" [][@-ln001 ~]$ srun -lN1 --environment=alpine.toml echo "abc" @@ -897,9 +916,11 @@ image = "alpine:3.19" 0: slurmstepd: error: pyxis: couldn't start container 0: slurmstepd: error: spank: required plugin spank_pyxis.so: task_init() failed with rc=-1 0: slurmstepd: error: Failed to invoke spank plugin stack +``` This is because some hooks (e.g., Slurm and CXI hooks) leverage ldconfig (from Glibc) when they bind-mount host libraries inside containers; since Alpine Linux provides an alternative ldconfig (from Musl Libc), it does not work as intended by hooks. As a workaround, users may disable problematic hooks. For example, +```bash [][@-ln001 ~]$ cat alpine_workaround.toml image = "alpine:3.19" [annotations] @@ -907,9 +928,6 @@ com.hooks.slurm.enabled = "false" com.hooks.cxi.enabled = "false" [][@-ln001 ~]$ srun -lN1 --environment=alpine_workaround.toml echo "abc" abc +``` Notice the section [annotations] disabling Slurm and CXI hooks. - -Further reading -Containerized use case examples -Building container images on Alps From e2390edf5f0628cfb9d588f836786bdb3b77d739 Mon Sep 17 00:00:00 2001 From: gwangmu <43980405+gwangmu@users.noreply.github.com> Date: Tue, 25 Feb 2025 11:38:53 +0100 Subject: [PATCH 04/34] Update mkdocs.yml --- mkdocs.yml | 1 + 1 file changed, 1 insertion(+) diff --git a/mkdocs.yml b/mkdocs.yml index ae677b77..7c2ecf4e 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -67,6 +67,7 @@ nav: - tools/index.md - 'slurm': tools/slurm.md - 'uenv': tools/uenv.md + - 'container-engine': tools/container-engine.md - 'Data Management and Storage': - storage/index.md - 'File Systems': storage/filesystems.md From 368b48e8087eecba5d117116a6d22c7aac03dbcd Mon Sep 17 00:00:00 2001 From: gwangmu <43980405+gwangmu@users.noreply.github.com> Date: Tue, 25 Feb 2025 11:41:56 +0100 Subject: [PATCH 05/34] Update container-engine.md --- docs/tools/container-engine.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/tools/container-engine.md b/docs/tools/container-engine.md index f5da13f8..4cb05dbd 100644 --- a/docs/tools/container-engine.md +++ b/docs/tools/container-engine.md @@ -7,7 +7,7 @@ The Container Engine (CE) toolset is designed to enable computing jobs to seamle Containers effectively encapsulate a software stack; however, to be useful in HPC computing environments, they often require the customization of bind mounts, environment variables, working directories, hooks, plugins, etc. To simplify this process, the Container Engine (CE) toolset supports the specification of user environments through Environment Definition Files. -An Environment Definition File (EDF) is a text file in the TOML format that declaratively and prescriptively represents the creation of a computing environment based on a container image. Users can create their own custom environments and share, edit, or build upon already existing environments. +An Environment Definition File (EDF) is a text file in the [TOML format](https://toml.io/en/) that declaratively and prescriptively represents the creation of a computing environment based on a container image. Users can create their own custom environments and share, edit, or build upon already existing environments. The Container Engine (CE) toolset leverages its tight integration with the Slurm workload manager to parse EDFs directly from the command line or batch script and instantiate containerized user environments seamlessly and transparently. @@ -33,7 +33,7 @@ workdir = "/capstor/scratch/cscs/${USER}" Note: Ensure that your ${USER} environment variable is defined with your actual username. -Save this file as ubuntu.toml file in $HOME/.edf directory (which is the default location of EDF files). A more detailed explanation of each entry for the EDF can be seen in the EDF reference. +Save this file as ubuntu.toml file in $HOME/.edf directory (which is the default location of EDF files). A more detailed explanation of each entry for the EDF can be seen in the [EDF reference](#edf-reference). ### Running the environment @@ -619,7 +619,7 @@ com.hooks.nvidia_cuda_mps.enabled = "true" > **INFO**: When using the NVIDIA CUDA MPS hook it is not necessary to use other wrappers or scripts to manage the Multi-Process Service, as is documented for native jobs on some vClusters. -## EDF Reference +## EDF Reference EDF files use the TOML format. For details about the data types used by the different parameters, please refer to the TOML spec webpage. From 1d4383dad12bd21907b704096692b9f964866f78 Mon Sep 17 00:00:00 2001 From: gwangmu <43980405+gwangmu@users.noreply.github.com> Date: Tue, 25 Feb 2025 11:44:26 +0100 Subject: [PATCH 06/34] Update container-engine.md --- docs/tools/container-engine.md | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/docs/tools/container-engine.md b/docs/tools/container-engine.md index 4cb05dbd..bc99cb35 100644 --- a/docs/tools/container-engine.md +++ b/docs/tools/container-engine.md @@ -43,7 +43,7 @@ Use Slurm in the cluster login node to start the Ubuntu environment that was jus $ srun --environment=ubuntu --pty bash ``` -Since the ubuntu.toml file is located in the EDF search path, the filename can be passed to the option without the file extension. +Since the ubuntu.toml file is located in the [EDF search path](#edf-search-path), the filename can be passed to the option without the file extension. ### Example Output @@ -113,7 +113,7 @@ SUPPORT_URL="https://www.debian.org/support" BUG_REPORT_URL="https://bugs.debian.org/" ``` -If a file is located in the EDF search path, the argument to the command line option can be just the environment name, that is the name of the file without the .toml extension, for example: +If a file is located in the [EDF search path](#edf-search-path), the argument to the command line option can be just the environment name, that is the name of the file without the .toml extension, for example: ```bash [][@-ln001 ~]$ srun --environment=debian cat /etc/os-release @@ -150,7 +150,7 @@ For the time being, if the script requires to invoke Slurm commands, the recomme srun --environment=debian cat /etc/os-release ``` -### The EDF search path +### The EDF search path By default, the EDFs for each user are looked up in $HOME/.edf. The search path for EDFs can be controlled through the EDF_PATH environment variable. EDF_PATH must be a colon-separated list of absolute paths to directories where the CE looks for TOML files, similar to the PATH and LD_LIBRARY_PATH variables. If a file is located in the search path, its name can be used in --environment options without the .toml extension, for example: @@ -341,7 +341,7 @@ To avoid mistakes, notice a few key features of TOML: * All property assignments belong to the section immediately preceding them (the statement in square brackets), which defines the table they refer to. * Tables, on the other hand, do not automatically belong to the tables declared before them; to nest tables, their name has to list their parents using the dot notations (so the previous example defines the table ssh inside hooks, which in turn is inside com, which is inside annotations). * An assignment can implicitly define subtables if the key you assign is a dotted list. As a reference, see the examples made earlier in this section, where assigning a string to the com.hooks.ssh.enabled attribute within the [annotations] table is exactly equivalent to assigning to the enabled attribute within the [annotations.com.hooks.ssh] subtable. - * Attributes can be added to a table only in one place in the TOML file. In other words, each table must be defined in a single square bracket section. For example, Case 3 in the example below is invalid because the ssh table is defined (gets attributes set) both in the [annotations] and in the [annotations.com.hooks.ssh] sections. See the TOML format spec for more details. + * Attributes can be added to a table only in one place in the TOML file. In other words, each table must be defined in a single square bracket section. For example, Case 3 in the example below is invalid because the ssh table is defined (gets attributes set) both in the [annotations] and in the [annotations.com.hooks.ssh] sections. See the [TOML format](https://toml.io/en/) spec for more details. * Case 1 (valid): ```bash [annotations.com.hooks.ssh] @@ -621,7 +621,7 @@ com.hooks.nvidia_cuda_mps.enabled = "true" ## EDF Reference -EDF files use the TOML format. For details about the data types used by the different parameters, please refer to the TOML spec webpage. +EDF files use the [TOML format](https://toml.io/en/). For details about the data types used by the different parameters, please refer to the TOML spec webpage. Parameter Description @@ -636,7 +636,7 @@ Default: N/A Ordered list of EDFs that this file inherits from. Parameters from listed environments are evaluated sequentially. Supports up to 10 levels of recursion. Parameters from the listed environments are evaluated sequentially, adding new entries or overwriting previous ones, before evaluating the parameters from the current EDF. In other words, the current EDF inherits the parameters from the EDFs listed in base_environment. When evaluating mounts or env parameters, values from downstream EDFs are appended to inherited values. -The individual EDF entries in the array follow the same search rules as the arguments of the --environment CLI option for Slurm; they can be either file paths or filenames without extension if the file is located in the EDF search path. +The individual EDF entries in the array follow the same search rules as the arguments of the --environment CLI option for Slurm; they can be either file paths or filenames without extension if the file is located in the [EDF search path](#edf-search-path). This parameter can be a string if there is only one base environment. From 98dac1b498999e0cecea431643ecf0f378e8478d Mon Sep 17 00:00:00 2001 From: gwangmu <43980405+gwangmu@users.noreply.github.com> Date: Tue, 25 Feb 2025 11:52:36 +0100 Subject: [PATCH 07/34] Update container-engine.md --- docs/tools/container-engine.md | 36 +++++++++++++++++----------------- 1 file changed, 18 insertions(+), 18 deletions(-) diff --git a/docs/tools/container-engine.md b/docs/tools/container-engine.md index bc99cb35..b7c6597b 100644 --- a/docs/tools/container-engine.md +++ b/docs/tools/container-engine.md @@ -131,7 +131,7 @@ BUG_REPORT_URL="https://bugs.debian.org/" ### Use from batch scripts In principle, the --environment option can also be used within batch scripts as an #SBATCH option. -It is important to note that in such a case, all the contents of the script are executed within the containerized environment: the CE toolset gives access to the Slurm workload manager within containers via the Slurm hook, see section Container Hooks (controlled by the ENROOT_SLURM_HOOK environment variable and activated by default on most vClusters). Only with it, calls to Slurm commands (for example srun or scontrol) within the batch script will work. +It is important to note that in such a case, all the contents of the script are executed within the containerized environment: the CE toolset gives access to the Slurm workload manager within containers via the Slurm hook, see section [Container Hooks](#container-hooks) (controlled by the ENROOT_SLURM_HOOK environment variable and activated by default on most vClusters). Only with it, calls to Slurm commands (for example srun or scontrol) within the batch script will work. For the time being, if the script requires to invoke Slurm commands, the recommended approach is to use --environment as part of the commands, for example, when launching job steps: @@ -264,7 +264,7 @@ image = "/capstor/scratch/cscs//nvidia+cuda+11.8.0-cudnn8-devel-ubuntu ### Third-party and private registries -Docker Hub is the default registry from which remote images are imported. +[Docker Hub](https://hub.docker.com/) is the default registry from which remote images are imported. To use an image from a different registry, the corresponding registry URL has to be prepended to the image reference, using a hash character (#) as a separator. For example: @@ -277,7 +277,7 @@ image = "nvcr.io#nvidia/nvhpc:23.7-runtime-cuda11.8-ubuntu22.04" [][@-ln001 ]$ srun enroot import docker://nvcr.io#nvidia/nvhpc:23.7-runtime-cuda11.8-ubuntu22.04 ``` -To import images from private repositories, access credentials should be configured by individual users in the $HOME/.config/enroot/.credentials file, following the netrc file format. +To import images from private repositories, access credentials should be configured by individual users in the $HOME/.config/enroot/.credentials file, following the [netrc file format](https://everything.curl.dev/usingcurl/netrc). Using the enroot import documentation page as a reference, some examples could be: ```bash @@ -318,9 +318,9 @@ machine gitlab.ethz.ch login password ## Annotations  -Annotations define arbitrary metadata for containers in the form of key-value pairs. Within the EDF, annotations are designed to be similar in appearance and behavior to those defined by the OCI Runtime Specification. Annotation keys usually express a hierarchical namespace structure, with domains separated by "." (full stop) characters. +Annotations define arbitrary metadata for containers in the form of key-value pairs. Within the EDF, annotations are designed to be similar in appearance and behavior to those defined by the [OCI Runtime Specification](https://github.com/opencontainers/runtime-spec/blob/main/config.md#annotations). Annotation keys usually express a hierarchical namespace structure, with domains separated by "." (full stop) characters. -As annotations are often used to control hooks, they have a deep nesting level. For example, to execute the SSH hook described below, the annotation com.hooks.ssh.enabled must be set to the string "true". +As annotations are often used to control hooks, they have a deep nesting level. For example, to execute the [SSH hook](#ssh-hook) described below, the annotation com.hooks.ssh.enabled must be set to the string "true". EDF files support setting annotations through the annotations table. This can be done in multiple ways in TOML: for example, both of the following usages are equivalent: @@ -412,16 +412,16 @@ Thu Oct 26 17:59:36 2023        It is possible to use environment variables to control which capabilities of the NVIDIA driver are enabled inside containers. Additionally, the NVIDIA Container Toolkit can enforce specific constraints for the container, for example, on versions of the CUDA runtime or driver, or on the architecture of the GPUs. -For the full details about using these features, please refer to the official documentation: Driver Capabilities, Constraints. +For the full details about using these features, please refer to the official documentation: [Driver Capabilities](https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/docker-specialized.html#driver-capabilities), [Constraints](https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/docker-specialized.html#constraints). -### HPE Slingshot interconnect  +### HPE Slingshot interconnect  -The Container Engine provides a hook to allow containers relying on libfabric to leverage the HPE Slingshot 11 high-speed interconnect. This component is commonly referred to as the "CXI hook", taking its name from the CXI libfabric provider required to interface with Slingshot 11. +The Container Engine provides a hook to allow containers relying on [libfabric](https://ofiwg.github.io/libfabric/) to leverage the HPE Slingshot 11 high-speed interconnect. This component is commonly referred to as the "CXI hook", taking its name from the CXI libfabric provider required to interface with Slingshot 11. The hook leverages bind-mounting the custom host libfabric library into the container (in addition to all the required dependency libraries and devices as well). If a libfabric library is already present in the container filesystem (for example, it's provided by the image), it is replaced with its host counterpart, otherwise the host libfabric is just added to the container. > **NOTE**: Due to the nature of Slingshot and the mechanism implemented by the CXI hook, container applications need to use a communication library which supports libfabric in order to benefit from usage of the hook. -> Libfabric support might have to be defined at compilation time (as is the case for some MPI implementations, like MPICH and OpenMPI) or could be dynamically available at runtime (as is the case with NCCL - see also this section for more details). +> Libfabric support might have to be defined at compilation time (as is the case for some MPI implementations, like MPICH and OpenMPI) or could be dynamically available at runtime (as is the case with NCCL - see also [this](#aws-ofi-hook) section for more details). The hook is activated by setting the com.hooks.cxi.enabled annotation, which can be defined in the EDF, as shown in the following example: @@ -498,7 +498,7 @@ com.hooks.cxi.enabled = "true" > **TIP**: On several vClusters, the CXI hook for Slingshot connectivity is enabled implicitly by default or by other hooks. Therefore, entering the enabling annotation in the EDF is unnecessary in many cases. -## Container Hooks +## Container Hooks Container hooks let you customize container behavior to fit system-specific needs, making them especially valuable for High-Performance Computing. @@ -507,9 +507,9 @@ Container hooks let you customize container behavior to fit system-specific need > **INFO**: This section outlines all hooks supported in production by the Container Engine. However, specific Alps vClusters may support only a subset or use custom configurations. For details about available features in individual vClusters, consult platform documentation or contact CSCS support. -### AWS OFI NCCL Hook  +### AWS OFI NCCL Hook  -The AWS OFI NCCL plugin is a software extension that allows the NCCL and RCCL libraries to use libfabric as a network provider and, through libfabric, to access the Slingshot high-speed interconnect. +The [AWS OFI NCCL plugin](https://github.com/aws/aws-ofi-nccl) is a software extension that allows the [NCCL](https://developer.nvidia.com/nccl) and [RCCL](https://rocm.docs.amd.com/projects/rccl/en/latest/) libraries to use libfabric as a network provider and, through libfabric, to access the Slingshot high-speed interconnect. The Container Engine includes a hook program to inject the AWS OFI NCCL plugin in containers; since the plugin must also be compatible with the GPU programming software stack being used, the com.hooks.aws_ofi_nccl.variant annotation is used to specify a plugin variant suitable for a given container image. At the moment of writing, 4 plugin variants are configured: cuda11, cuda12 (to be used on NVIDIA GPU nodes), rocm5, and rocm6 (to be used on AMD GPU nodes alongside RCCL). @@ -527,12 +527,12 @@ com.hooks.aws_ofi_nccl.variant = "cuda11" The AWS OFI NCCL hook also takes care of the following aspects: - * It implicitly enables the CXI hook, therefore exposing the Slingshot interconnect to container applications. In other words, when enabling the AWS OFI NCCL hook, it's unnecessary to also enable the CXI hook separately in the EDF. + * It implicitly enables the [CXI hook](#cxi-hook), therefore exposing the Slingshot interconnect to container applications. In other words, when enabling the AWS OFI NCCL hook, it's unnecessary to also enable the CXI hook separately in the EDF. * It sets environment variables to control the behavior of NCCL and the libfabric CXI provider for Slingshot. In particular, the NCCL_NET_PLUGIN variable is set to force NCCL to load the specific network plugin mounted by the hook. This is useful because certain container images (for example, those from NGC repositories) might already ship with a default NCCL plugin. Other environment variables help prevent application stalls and improve performance when using GPUDirect for RDMA communication. -### SSH Hook +### SSH Hook -The SSH hook runs a lightweight, statically-linked SSH server (a build of Dropbear) inside the container. It can be useful to add SSH connectivity to containers (for example, enabling remote debugging) without bundling an SSH server into the container image or creating ad-hoc image variants for such purposes. +The SSH hook runs a lightweight, statically-linked SSH server (a build of [Dropbear](https://matt.ucc.asn.au/dropbear/dropbear.html)) inside the container. It can be useful to add SSH connectivity to containers (for example, enabling remote debugging) without bundling an SSH server into the container image or creating ad-hoc image variants for such purposes. The com.hooks.ssh.authorize_ssh_key annotation allows the authorization of a custom public SSH key for remote connections. The annotation value must be the absolute path to a text file containing the public key (just the public key without any extra signature/certificate). After the container starts, it is possible to get a remote shell inside the container by connecting with SSH to the listening port. @@ -564,7 +564,7 @@ While the container is running, it's possible to connect to it from a remote hos ssh -p 15263 ``` -> **INFO**: In order to establish connections through Visual Studio Code Remote - SSH extension, the scp program must be available within the container. This is required to send and establish the VS Code Server into the remote container. +> **INFO**: In order to establish connections through Visual Studio Code [Remote - SSH](https://code.visualstudio.com/docs/remote/ssh) extension, the scp program must be available within the container. This is required to send and establish the VS Code Server into the remote container. ### NVIDIA CUDA MPS Hook @@ -596,7 +596,7 @@ srun: error: [...] [...] ``` -In order to run multiple processes concurrently on the same GPU (one example could be running multiple MPI ranks on the same device), the NVIDIA CUDA Multi-Process Service (or MPS, for short) must be started on the compute node. +In order to run multiple processes concurrently on the same GPU (one example could be running multiple MPI ranks on the same device), the [NVIDIA CUDA Multi-Process Service](https://docs.nvidia.com/deploy/mps/index.html) (or MPS, for short) must be started on the compute node. The Container Engine provides a hook to automatically manage the setup and removal of the NVIDIA CUDA MPS components within containers. The hook can be activated by setting the com.hooks.nvidia_cuda_mps.enabled to the string true. @@ -621,7 +621,7 @@ com.hooks.nvidia_cuda_mps.enabled = "true" ## EDF Reference -EDF files use the [TOML format](https://toml.io/en/). For details about the data types used by the different parameters, please refer to the TOML spec webpage. +EDF files use the [TOML format](https://toml.io/en/). For details about the data types used by the different parameters, please refer to the [TOML spec webpage](https://toml.io/en/v1.0.0). Parameter Description From 45e2106411909defe26ce233a02effe0fe59a306 Mon Sep 17 00:00:00 2001 From: gwangmu <43980405+gwangmu@users.noreply.github.com> Date: Tue, 25 Feb 2025 12:46:08 +0100 Subject: [PATCH 08/34] Update container-engine.md --- docs/tools/container-engine.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/docs/tools/container-engine.md b/docs/tools/container-engine.md index b7c6597b..7fb2b9b6 100644 --- a/docs/tools/container-engine.md +++ b/docs/tools/container-engine.md @@ -31,9 +31,9 @@ mounts = ["/capstor/scratch/cscs/${USER}:/capstor/scratch/cscs/${USER}"] workdir = "/capstor/scratch/cscs/${USER}" ``` -Note: Ensure that your ${USER} environment variable is defined with your actual username. +Note: Ensure that your `${USER}` environment variable is defined with your actual username. -Save this file as ubuntu.toml file in $HOME/.edf directory (which is the default location of EDF files). A more detailed explanation of each entry for the EDF can be seen in the [EDF reference](#edf-reference). +Save this file as `ubuntu.toml` file in `$HOME/.edf` directory (which is the default location of EDF files). A more detailed explanation of each entry for the EDF can be seen in the [EDF reference](#edf-reference). ### Running the environment @@ -43,7 +43,7 @@ Use Slurm in the cluster login node to start the Ubuntu environment that was jus $ srun --environment=ubuntu --pty bash ``` -Since the ubuntu.toml file is located in the [EDF search path](#edf-search-path), the filename can be passed to the option without the file extension. +Since the `ubuntu.toml` file is located in the [EDF search path](#edf-search-path), the filename can be passed to the option without the file extension. ### Example Output @@ -72,7 +72,7 @@ LOGO=ubuntu-logo [][@-ln001 ~]$ ``` -The above terminal snippet demonstrates how to launch a containerized environment using Slurm with the --environment option, where we highlight: +The above terminal snippet demonstrates how to launch a containerized environment using Slurm with the `--environment` option, where we highlight: Step 1. Starting an interactive shell session within the Ubuntu 24.04 container deployed on a compute node using srun --environment=ubuntu --pty bash. Step 2. Confirm the working directory inside the container (pwd) and that it is set to the user's scratch folder, as per EDF. From efd991d442c828a0c977a597a4f77e1ea7f0c87e Mon Sep 17 00:00:00 2001 From: Gwangmu Lee Date: Tue, 25 Feb 2025 13:11:21 +0100 Subject: [PATCH 09/34] Monospace-ize some words --- docs/tools/container-engine.md | 70 +++++++++++++++++----------------- 1 file changed, 35 insertions(+), 35 deletions(-) diff --git a/docs/tools/container-engine.md b/docs/tools/container-engine.md index 7fb2b9b6..d93ca432 100644 --- a/docs/tools/container-engine.md +++ b/docs/tools/container-engine.md @@ -74,16 +74,16 @@ LOGO=ubuntu-logo The above terminal snippet demonstrates how to launch a containerized environment using Slurm with the `--environment` option, where we highlight: - Step 1. Starting an interactive shell session within the Ubuntu 24.04 container deployed on a compute node using srun --environment=ubuntu --pty bash. - Step 2. Confirm the working directory inside the container (pwd) and that it is set to the user's scratch folder, as per EDF. - Step 3. Show the OS version of your container (using cat /etc/os-release) based on Ubuntu 24.04 LTS. - Step 4. Exiting the container (exit), returning to the login node. + Step 1. Starting an interactive shell session within the Ubuntu 24.04 container deployed on a compute node using `srun --environment=ubuntu --pty bash`. + Step 2. Confirm the working directory inside the container (`pwd`) and that it is set to the user's scratch folder, as per EDF. + Step 3. Show the OS version of your container (using `cat /etc/os-release`) based on Ubuntu 24.04 LTS. + Step 4. Exiting the container (`exit`), returning to the login node. Note that the image pull and the container start happen automatically, streamlining the usage of the CE. ## Running containerized environments -A job is run in a containerized environment by passing the --environment option to the srun or salloc Slurm commands. The option takes a file path to the EDF describing the environment in which the job should be executed, for example: +A job is run in a containerized environment by passing the `--environment` option to the `srun` or `salloc` Slurm commands. The option takes a file path to the EDF describing the environment in which the job should be executed, for example: ```bash [][@-ln001 ~]$ srun --environment=$SCRATCH/edf/debian.toml cat /etc/os-release @@ -97,7 +97,7 @@ HOME_URL="https://www.debian.org/" SUPPORT_URL="https://www.debian.org/support" BUG_REPORT_URL="https://bugs.debian.org/" ---environment can be a relative path from the current working directory (i.e., where the Slurm command is entered). A relative path should be prepended by ./. For example: +`--environment` can be a relative path from the current working directory (i.e., where the Slurm command is entered). A relative path should be prepended by `./`. For example: [][@-ln001 ~]$ ls debian.toml @@ -113,7 +113,7 @@ SUPPORT_URL="https://www.debian.org/support" BUG_REPORT_URL="https://bugs.debian.org/" ``` -If a file is located in the [EDF search path](#edf-search-path), the argument to the command line option can be just the environment name, that is the name of the file without the .toml extension, for example: +If a file is located in the [EDF search path](#edf-search-path), the argument to the command line option can be just the environment name, that is the name of the file without the `.toml` extension, for example: ```bash [][@-ln001 ~]$ srun --environment=debian cat /etc/os-release @@ -130,11 +130,11 @@ BUG_REPORT_URL="https://bugs.debian.org/" ### Use from batch scripts -In principle, the --environment option can also be used within batch scripts as an #SBATCH option. -It is important to note that in such a case, all the contents of the script are executed within the containerized environment: the CE toolset gives access to the Slurm workload manager within containers via the Slurm hook, see section [Container Hooks](#container-hooks) (controlled by the ENROOT_SLURM_HOOK environment variable and activated by default on most vClusters). Only with it, calls to Slurm commands (for example srun or scontrol) within the batch script will work. +In principle, the `--environment` option can also be used within batch scripts as an `#SBATCH` option. +It is important to note that in such a case, all the contents of the script are executed within the containerized environment: the CE toolset gives access to the Slurm workload manager within containers via the Slurm hook, see section [Container Hooks](#container-hooks) (controlled by the `ENROOT_SLURM_HOOK` environment variable and activated by default on most vClusters). Only with it, calls to Slurm commands (for example `srun` or `scontrol`) within the batch script will work. -For the time being, if the script requires to invoke Slurm commands, the recommended approach is to use --environment as part of the commands, for example, when launching job steps: +For the time being, if the script requires to invoke Slurm commands, the recommended approach is to use `--environment` as part of the commands, for example, when launching job steps: ``` [][@-ln001 ~]$ cat example.sbatch @@ -152,7 +152,7 @@ srun --environment=debian cat /etc/os-release ### The EDF search path -By default, the EDFs for each user are looked up in $HOME/.edf. The search path for EDFs can be controlled through the EDF_PATH environment variable. EDF_PATH must be a colon-separated list of absolute paths to directories where the CE looks for TOML files, similar to the PATH and LD_LIBRARY_PATH variables. If a file is located in the search path, its name can be used in --environment options without the .toml extension, for example: +By default, the EDFs for each user are looked up in `$HOME/.edf`. The search path for EDFs can be controlled through the `EDF_PATH` environment variable. `EDF_PATH` must be a colon-separated list of absolute paths to directories where the CE looks for TOML files, similar to the `PATH` and L`D_LIBRARY_PATH` variables. If a file is located in the search path, its name can be used in `--environment` options without the `.toml` extension, for example: ```bash [][@-ln001 ~]$ ls -l ~/.edf @@ -184,11 +184,11 @@ PRETTY_NAME="Fedora Linux 40 (Container Image)" By default, images defined in the EDF as remote registry references (e.g. a Docker reference) are automatically pulled and locally cached. A cached image would be preferred to pulling the image again in later usage.  -An image cache is automatically created at .edf_imagestore in the user's scratch folder (i.e., $SCRATCH/.edf_imagestore), under which cached images are stored in a corresponding CPU architecture subfolder (e.g., x86 and aarch64 ). Users should regularly remove unused cache images to limit the cache size. +An image cache is automatically created at `.edf_imagestore` in the user's scratch folder (i.e., `$SCRATCH/.edf_imagestore`), under which cached images are stored in a corresponding CPU architecture subfolder (e.g., `x86` and `aarch64`). Users should regularly remove unused cache images to limit the cache size. Should users want to re-pull a cached image, they have to remove the corresponding image in the cache. -To choose an alternative image store path (e.g., to use a directory owned by a group and not to an individual user), users can specify an image cache path explicitly by defining the environment variable EDF_IMAGESTORE. EDF_IMAGESTORE must be an absolute path to an existing folder. +To choose an alternative image store path (e.g., to use a directory owned by a group and not to an individual user), users can specify an image cache path explicitly by defining the environment variable `EDF_IMAGESTORE`. `EDF_IMAGESTORE` must be an absolute path to an existing folder. > **NOTE**: If the CE cannot create a directory for the image cache, it operates in cache-free mode, meaning that it pulls an ephemeral image before every container launch and discards it upon termination. @@ -196,9 +196,9 @@ To choose an alternative image store path (e.g., to use a directory owned by a g To work with images stored from the NGC Catalog, please refer also to the next section "Using images from third party registries and private repositories". -To bypass any caching behavior, users can manually pull an image and directly plug it into their EDF. To do so, users may execute enroot import docker://[REGISTRY#]IMAGE[:TAG] to pull container images from OCI registries to the current directory. +To bypass any caching behavior, users can manually pull an image and directly plug it into their EDF. To do so, users may execute `enroot import docker://[REGISTRY#]IMAGE[:TAG]` to pull container images from OCI registries to the current directory. -For example, the command below pulls an nvidia/cuda:11.8.0-cudnn8-devel-ubuntu22.04 image. +For example, the command below pulls an `nvidia/cuda:11.8.0-cudnn8-devel-ubuntu22.04` image. ```bash $ enroot import docker://nvidia/cuda:11.8.0-cudnn8-devel-ubuntu22.04 @@ -260,7 +260,7 @@ After the import is complete, images are available in Squashfs format in the cur image = "/capstor/scratch/cscs//nvidia+cuda+11.8.0-cudnn8-devel-ubuntu22.04.sqsh" ``` -> **NOTE**: It is recommended to save images in /capstor/scratch/cscs/ or its subdirectories before using them with the CE. +> **NOTE**: It is recommended to save images in `/capstor/scratch/cscs/` or its subdirectories before using them with the CE. ### Third-party and private registries @@ -277,8 +277,8 @@ image = "nvcr.io#nvidia/nvhpc:23.7-runtime-cuda11.8-ubuntu22.04" [][@-ln001 ]$ srun enroot import docker://nvcr.io#nvidia/nvhpc:23.7-runtime-cuda11.8-ubuntu22.04 ``` -To import images from private repositories, access credentials should be configured by individual users in the $HOME/.config/enroot/.credentials file, following the [netrc file format](https://everything.curl.dev/usingcurl/netrc). -Using the enroot import documentation page as a reference, some examples could be: +To import images from private repositories, access credentials should be configured by individual users in the `$HOME/.config/enroot/.credentials` file, following the [netrc file format](https://everything.curl.dev/usingcurl/netrc). +Using the `enroot import` documentation page as a reference, some examples could be: ```bash # NVIDIA NGC catalog (both endpoints are required) @@ -320,9 +320,9 @@ machine gitlab.ethz.ch login password Annotations define arbitrary metadata for containers in the form of key-value pairs. Within the EDF, annotations are designed to be similar in appearance and behavior to those defined by the [OCI Runtime Specification](https://github.com/opencontainers/runtime-spec/blob/main/config.md#annotations). Annotation keys usually express a hierarchical namespace structure, with domains separated by "." (full stop) characters. -As annotations are often used to control hooks, they have a deep nesting level. For example, to execute the [SSH hook](#ssh-hook) described below, the annotation com.hooks.ssh.enabled must be set to the string "true". +As annotations are often used to control hooks, they have a deep nesting level. For example, to execute the [SSH hook](#ssh-hook) described below, the annotation `com.hooks.ssh.enabled` must be set to the string `true`. -EDF files support setting annotations through the annotations table. This can be done in multiple ways in TOML: for example, both of the following usages are equivalent: +EDF files support setting annotations through the `annotations` table. This can be done in multiple ways in TOML: for example, both of the following usages are equivalent: * Case: nest levels in the TOML key. ```bash @@ -339,9 +339,9 @@ enabled = "true" To avoid mistakes, notice a few key features of TOML: * All property assignments belong to the section immediately preceding them (the statement in square brackets), which defines the table they refer to. - * Tables, on the other hand, do not automatically belong to the tables declared before them; to nest tables, their name has to list their parents using the dot notations (so the previous example defines the table ssh inside hooks, which in turn is inside com, which is inside annotations). - * An assignment can implicitly define subtables if the key you assign is a dotted list. As a reference, see the examples made earlier in this section, where assigning a string to the com.hooks.ssh.enabled attribute within the [annotations] table is exactly equivalent to assigning to the enabled attribute within the [annotations.com.hooks.ssh] subtable. - * Attributes can be added to a table only in one place in the TOML file. In other words, each table must be defined in a single square bracket section. For example, Case 3 in the example below is invalid because the ssh table is defined (gets attributes set) both in the [annotations] and in the [annotations.com.hooks.ssh] sections. See the [TOML format](https://toml.io/en/) spec for more details. + * Tables, on the other hand, do not automatically belong to the tables declared before them; to nest tables, their name has to list their parents using the dot notations (so the previous example defines the table `ssh` inside `hooks`, which in turn is inside `com`, which is inside `annotations`). + * An assignment can implicitly define subtables if the key you assign is a dotted list. As a reference, see the examples made earlier in this section, where assigning a string to the `com.hooks.ssh.enabled` attribute within the `[annotations]` table is exactly equivalent to assigning to the `enabled` attribute within the `[annotations.com.hooks.ssh]` subtable. + * Attributes can be added to a table only in one place in the TOML file. In other words, each table must be defined in a single square bracket section. For example, Case 3 in the example below is invalid because the `ssh` table is defined (gets attributes set) both in the `[annotations]` and in the `[annotations.com.hooks.ssh]` sections. See the [TOML format](https://toml.io/en/) spec for more details. * Case 1 (valid): ```bash [annotations.com.hooks.ssh] @@ -368,7 +368,7 @@ enabled = "true" ### NVIDIA GPUs The Container Engine leverages components from the NVIDIA Container Toolkit to expose NVIDIA GPU devices inside containers. -GPU device files are always mounted in containers, and the NVIDIA driver user space components are  mounted if the NVIDIA_VISIBLE_DEVICES environment variable is not empty, unset or set to "void".  NVIDIA_VISIBLE_DEVICES is already set in container images officially provided by NVIDIA to enable all GPUs available on the host system. Such images are frequently used to containerize CUDA applications, either directly or as a base for custom images, thus in many cases no action is required to access GPUs. +GPU device files are always mounted in containers, and the NVIDIA driver user space components are  mounted if the `NVIDIA_VISIBLE_DEVICES` environment variable is not empty, unset or set to `void`.  `NVIDIA_VISIBLE_DEVICES` is already set in container images officially provided by NVIDIA to enable all GPUs available on the host system. Such images are frequently used to containerize CUDA applications, either directly or as a base for custom images, thus in many cases no action is required to access GPUs. For example, on a cluster with 4 GH200 devices per compute node: ```bash @@ -423,7 +423,7 @@ If a libfabric library is already present in the container filesystem (for examp > **NOTE**: Due to the nature of Slingshot and the mechanism implemented by the CXI hook, container applications need to use a communication library which supports libfabric in order to benefit from usage of the hook. > Libfabric support might have to be defined at compilation time (as is the case for some MPI implementations, like MPICH and OpenMPI) or could be dynamically available at runtime (as is the case with NCCL - see also [this](#aws-ofi-hook) section for more details). -The hook is activated by setting the com.hooks.cxi.enabled annotation, which can be defined in the EDF, as shown in the following example: +The hook is activated by setting the `com.hooks.cxi.enabled` annotation, which can be defined in the EDF, as shown in the following example: ```bash # Without the CXI hook @@ -511,8 +511,8 @@ Container hooks let you customize container behavior to fit system-specific need The [AWS OFI NCCL plugin](https://github.com/aws/aws-ofi-nccl) is a software extension that allows the [NCCL](https://developer.nvidia.com/nccl) and [RCCL](https://rocm.docs.amd.com/projects/rccl/en/latest/) libraries to use libfabric as a network provider and, through libfabric, to access the Slingshot high-speed interconnect. -The Container Engine includes a hook program to inject the AWS OFI NCCL plugin in containers; since the plugin must also be compatible with the GPU programming software stack being used, the com.hooks.aws_ofi_nccl.variant annotation is used to specify a plugin variant suitable for a given container image. -At the moment of writing, 4 plugin variants are configured: cuda11, cuda12 (to be used on NVIDIA GPU nodes), rocm5, and rocm6 (to be used on AMD GPU nodes alongside RCCL). +The Container Engine includes a hook program to inject the AWS OFI NCCL plugin in containers; since the plugin must also be compatible with the GPU programming software stack being used, the `com.hooks.aws_ofi_nccl.variant` annotation is used to specify a plugin variant suitable for a given container image. +At the moment of writing, 4 plugin variants are configured: `cuda11`, `cuda12` (to be used on NVIDIA GPU nodes), `rocm5`, and `rocm6` (to be used on AMD GPU nodes alongside RCCL). For example, the following EDF enables the hook and uses it to mount the plugin in a CUDA 11 image: ```bash @@ -528,15 +528,15 @@ com.hooks.aws_ofi_nccl.variant = "cuda11" The AWS OFI NCCL hook also takes care of the following aspects: * It implicitly enables the [CXI hook](#cxi-hook), therefore exposing the Slingshot interconnect to container applications. In other words, when enabling the AWS OFI NCCL hook, it's unnecessary to also enable the CXI hook separately in the EDF. - * It sets environment variables to control the behavior of NCCL and the libfabric CXI provider for Slingshot. In particular, the NCCL_NET_PLUGIN variable is set to force NCCL to load the specific network plugin mounted by the hook. This is useful because certain container images (for example, those from NGC repositories) might already ship with a default NCCL plugin. Other environment variables help prevent application stalls and improve performance when using GPUDirect for RDMA communication. + * It sets environment variables to control the behavior of NCCL and the libfabric CXI provider for Slingshot. In particular, the `NCCL_NET_PLUGIN` variable ([link](https://docs.nvidia.com/deeplearning/nccl/user-guide/docs/env.html#nccl-net-plugin)) is set to force NCCL to load the specific network plugin mounted by the hook. This is useful because certain container images (for example, those from NGC repositories) might already ship with a default NCCL plugin. Other environment variables help prevent application stalls and improve performance when using GPUDirect for RDMA communication. ### SSH Hook The SSH hook runs a lightweight, statically-linked SSH server (a build of [Dropbear](https://matt.ucc.asn.au/dropbear/dropbear.html)) inside the container. It can be useful to add SSH connectivity to containers (for example, enabling remote debugging) without bundling an SSH server into the container image or creating ad-hoc image variants for such purposes. -The com.hooks.ssh.authorize_ssh_key annotation allows the authorization of a custom public SSH key for remote connections. The annotation value must be the absolute path to a text file containing the public key (just the public key without any extra signature/certificate). After the container starts, it is possible to get a remote shell inside the container by connecting with SSH to the listening port. +The `com.hooks.ssh.authorize_ssh_key` annotation allows the authorization of a custom public SSH key for remote connections. The annotation value must be the absolute path to a text file containing the public key (just the public key without any extra signature/certificate). After the container starts, it is possible to get a remote shell inside the container by connecting with SSH to the listening port. -By default, the server started by the SSH hook listens to port 15263, but this setting can be controlled through the com.hooks.ssh.port annotation in the EDF. +By default, the server started by the SSH hook listens to port 15263, but this setting can be controlled through the `com.hooks.ssh.port` annotation in the EDF. > **NOTE**: To use the SSH hook, it is **required** to keep the container **writable**. @@ -552,7 +552,7 @@ enabled = "true" authorize_ssh_key = "" ``` -Using the previous EDF, a container can be started as follows. Notice that the --pty option for the srun command is currently required in order for the hook to initialize properly: +Using the previous EDF, a container can be started as follows. Notice that the `--pty` option for the `srun` command is currently required in order for the hook to initialize properly: ```bash [][@-ln001 ~]$ srun --environment=ubuntu-ssh --pty @@ -564,7 +564,7 @@ While the container is running, it's possible to connect to it from a remote hos ssh -p 15263 ``` -> **INFO**: In order to establish connections through Visual Studio Code [Remote - SSH](https://code.visualstudio.com/docs/remote/ssh) extension, the scp program must be available within the container. This is required to send and establish the VS Code Server into the remote container. +> **INFO**: In order to establish connections through Visual Studio Code [Remote - SSH](https://code.visualstudio.com/docs/remote/ssh) extension, the `scp` program must be available within the container. This is required to send and establish the VS Code Server into the remote container. ### NVIDIA CUDA MPS Hook @@ -599,7 +599,7 @@ srun: error: [...] In order to run multiple processes concurrently on the same GPU (one example could be running multiple MPI ranks on the same device), the [NVIDIA CUDA Multi-Process Service](https://docs.nvidia.com/deploy/mps/index.html) (or MPS, for short) must be started on the compute node. The Container Engine provides a hook to automatically manage the setup and removal of the NVIDIA CUDA MPS components within containers. -The hook can be activated by setting the com.hooks.nvidia_cuda_mps.enabled to the string true. +The hook can be activated by setting the `com.hooks.nvidia_cuda_mps.enabled` to the string `true`. > **NOTE**: To use the CUDA MPS hook, it is **required** to keep the container **writable**. @@ -918,7 +918,7 @@ image = "alpine:3.19" 0: slurmstepd: error: Failed to invoke spank plugin stack ``` -This is because some hooks (e.g., Slurm and CXI hooks) leverage ldconfig (from Glibc) when they bind-mount host libraries inside containers; since Alpine Linux provides an alternative ldconfig (from Musl Libc), it does not work as intended by hooks. As a workaround, users may disable problematic hooks. For example, +This is because some hooks (e.g., Slurm and CXI hooks) leverage `ldconfig` (from Glibc) when they bind-mount host libraries inside containers; since Alpine Linux provides an alternative `ldconfig` (from Musl Libc), it does not work as intended by hooks. As a workaround, users may disable problematic hooks. For example, ```bash [][@-ln001 ~]$ cat alpine_workaround.toml @@ -930,4 +930,4 @@ com.hooks.cxi.enabled = "false" abc ``` -Notice the section [annotations] disabling Slurm and CXI hooks. +Notice the section `[annotations]` disabling Slurm and CXI hooks. From 4cbba961a7589eb184d0e2fa2070c0a45709563c Mon Sep 17 00:00:00 2001 From: Gwangmu Lee Date: Tue, 25 Feb 2025 13:12:42 +0100 Subject: [PATCH 10/34] Monospace-ize some words --- docs/tools/container-engine.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/tools/container-engine.md b/docs/tools/container-engine.md index d93ca432..4a77a179 100644 --- a/docs/tools/container-engine.md +++ b/docs/tools/container-engine.md @@ -136,7 +136,7 @@ It is important to note that in such a case, all the contents of the script are For the time being, if the script requires to invoke Slurm commands, the recommended approach is to use `--environment` as part of the commands, for example, when launching job steps: -``` +```bash [][@-ln001 ~]$ cat example.sbatch #!/bin/bash -l #SBATCH --job-name=edf-example From 9f16580b5357cf359b264fc3eea5e0af81efdc3e Mon Sep 17 00:00:00 2001 From: Gwangmu Lee Date: Tue, 25 Feb 2025 13:13:52 +0100 Subject: [PATCH 11/34] Monospace-ize some words --- docs/tools/container-engine.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/docs/tools/container-engine.md b/docs/tools/container-engine.md index 4a77a179..e5e9912d 100644 --- a/docs/tools/container-engine.md +++ b/docs/tools/container-engine.md @@ -348,12 +348,14 @@ To avoid mistakes, notice a few key features of TOML: authorize_ssh_key = "/capstor/scratch/cscs//tests/edf/authorized_keys" enabled = "true" ``` + * Case 2 (valid): ```bash [annotations] com.hooks.ssh.authorize_ssh_key = "/capstor/scratch/cscs//tests/edf/authorized_keys" com.hooks.ssh.enabled = "true" ``` + * Case 3 (**invalid**): ```bash [annotations] From 5a14d228ec117bd2c7b6af9115d39600719bd483 Mon Sep 17 00:00:00 2001 From: Gwangmu Lee Date: Tue, 25 Feb 2025 13:15:31 +0100 Subject: [PATCH 12/34] Monospace-ize some words --- docs/tools/container-engine.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/tools/container-engine.md b/docs/tools/container-engine.md index e5e9912d..b937e5a2 100644 --- a/docs/tools/container-engine.md +++ b/docs/tools/container-engine.md @@ -341,7 +341,7 @@ To avoid mistakes, notice a few key features of TOML: * All property assignments belong to the section immediately preceding them (the statement in square brackets), which defines the table they refer to. * Tables, on the other hand, do not automatically belong to the tables declared before them; to nest tables, their name has to list their parents using the dot notations (so the previous example defines the table `ssh` inside `hooks`, which in turn is inside `com`, which is inside `annotations`). * An assignment can implicitly define subtables if the key you assign is a dotted list. As a reference, see the examples made earlier in this section, where assigning a string to the `com.hooks.ssh.enabled` attribute within the `[annotations]` table is exactly equivalent to assigning to the `enabled` attribute within the `[annotations.com.hooks.ssh]` subtable. - * Attributes can be added to a table only in one place in the TOML file. In other words, each table must be defined in a single square bracket section. For example, Case 3 in the example below is invalid because the `ssh` table is defined (gets attributes set) both in the `[annotations]` and in the `[annotations.com.hooks.ssh]` sections. See the [TOML format](https://toml.io/en/) spec for more details. + * Attributes can be added to a table only in one place in the TOML file. In other words, each table must be defined in a single square bracket section. For example, Case 3 in the example below is invalid because the `ssh` table was defined both in the `[annotations]` and in the `[annotations.com.hooks.ssh]` sections. See the [TOML format](https://toml.io/en/) spec for more details. * Case 1 (valid): ```bash [annotations.com.hooks.ssh] From 8324ee6b67cdb3c41acbd588887581462646f3b0 Mon Sep 17 00:00:00 2001 From: Gwangmu Lee Date: Tue, 25 Feb 2025 13:16:12 +0100 Subject: [PATCH 13/34] Monospace-ize some words --- docs/tools/container-engine.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/tools/container-engine.md b/docs/tools/container-engine.md index b937e5a2..28157b70 100644 --- a/docs/tools/container-engine.md +++ b/docs/tools/container-engine.md @@ -341,7 +341,7 @@ To avoid mistakes, notice a few key features of TOML: * All property assignments belong to the section immediately preceding them (the statement in square brackets), which defines the table they refer to. * Tables, on the other hand, do not automatically belong to the tables declared before them; to nest tables, their name has to list their parents using the dot notations (so the previous example defines the table `ssh` inside `hooks`, which in turn is inside `com`, which is inside `annotations`). * An assignment can implicitly define subtables if the key you assign is a dotted list. As a reference, see the examples made earlier in this section, where assigning a string to the `com.hooks.ssh.enabled` attribute within the `[annotations]` table is exactly equivalent to assigning to the `enabled` attribute within the `[annotations.com.hooks.ssh]` subtable. - * Attributes can be added to a table only in one place in the TOML file. In other words, each table must be defined in a single square bracket section. For example, Case 3 in the example below is invalid because the `ssh` table was defined both in the `[annotations]` and in the `[annotations.com.hooks.ssh]` sections. See the [TOML format](https://toml.io/en/) spec for more details. + * Attributes can be added to a table only in one place in the TOML file. In other words, each table must be defined in a single square bracket section. For example, Case 3 in the example below is invalid because the `ssh` table was doubly defined both in the `[annotations]` and in the `[annotations.com.hooks.ssh]` sections. See the [TOML format](https://toml.io/en/) spec for more details. * Case 1 (valid): ```bash [annotations.com.hooks.ssh] From ab9674736c15a91efc4fa330392ecb11380aaee5 Mon Sep 17 00:00:00 2001 From: Gwangmu Lee Date: Tue, 25 Feb 2025 13:17:12 +0100 Subject: [PATCH 14/34] Monospace-ize some words --- docs/tools/container-engine.md | 32 ++++++++++++++++---------------- 1 file changed, 16 insertions(+), 16 deletions(-) diff --git a/docs/tools/container-engine.md b/docs/tools/container-engine.md index 28157b70..86a82cb0 100644 --- a/docs/tools/container-engine.md +++ b/docs/tools/container-engine.md @@ -343,27 +343,27 @@ To avoid mistakes, notice a few key features of TOML: * An assignment can implicitly define subtables if the key you assign is a dotted list. As a reference, see the examples made earlier in this section, where assigning a string to the `com.hooks.ssh.enabled` attribute within the `[annotations]` table is exactly equivalent to assigning to the `enabled` attribute within the `[annotations.com.hooks.ssh]` subtable. * Attributes can be added to a table only in one place in the TOML file. In other words, each table must be defined in a single square bracket section. For example, Case 3 in the example below is invalid because the `ssh` table was doubly defined both in the `[annotations]` and in the `[annotations.com.hooks.ssh]` sections. See the [TOML format](https://toml.io/en/) spec for more details. * Case 1 (valid): -```bash -[annotations.com.hooks.ssh] -authorize_ssh_key = "/capstor/scratch/cscs//tests/edf/authorized_keys" -enabled = "true" -``` + ```bash + [annotations.com.hooks.ssh] + authorize_ssh_key = "/capstor/scratch/cscs//tests/edf/authorized_keys" + enabled = "true" + ``` * Case 2 (valid): -```bash -[annotations] -com.hooks.ssh.authorize_ssh_key = "/capstor/scratch/cscs//tests/edf/authorized_keys" -com.hooks.ssh.enabled = "true" -``` + ```bash + [annotations] + com.hooks.ssh.authorize_ssh_key = "/capstor/scratch/cscs//tests/edf/authorized_keys" + com.hooks.ssh.enabled = "true" + ``` * Case 3 (**invalid**): -```bash -[annotations] -com.hooks.ssh.authorize_ssh_key = "/capstor/scratch/cscs//tests/edf/authorized_keys" + ```bash + [annotations] + com.hooks.ssh.authorize_ssh_key = "/capstor/scratch/cscs//tests/edf/authorized_keys" -[annotations.com.hooks.ssh] -enabled = "true" -``` + [annotations.com.hooks.ssh] + enabled = "true" + ``` ## Accessing native resources From 584ea8521f61333e4d67916716fc83257ddbad82 Mon Sep 17 00:00:00 2001 From: Gwangmu Lee Date: Tue, 25 Feb 2025 13:17:34 +0100 Subject: [PATCH 15/34] Monospace-ize some words --- docs/tools/container-engine.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/docs/tools/container-engine.md b/docs/tools/container-engine.md index 86a82cb0..6f1edfb4 100644 --- a/docs/tools/container-engine.md +++ b/docs/tools/container-engine.md @@ -343,11 +343,11 @@ To avoid mistakes, notice a few key features of TOML: * An assignment can implicitly define subtables if the key you assign is a dotted list. As a reference, see the examples made earlier in this section, where assigning a string to the `com.hooks.ssh.enabled` attribute within the `[annotations]` table is exactly equivalent to assigning to the `enabled` attribute within the `[annotations.com.hooks.ssh]` subtable. * Attributes can be added to a table only in one place in the TOML file. In other words, each table must be defined in a single square bracket section. For example, Case 3 in the example below is invalid because the `ssh` table was doubly defined both in the `[annotations]` and in the `[annotations.com.hooks.ssh]` sections. See the [TOML format](https://toml.io/en/) spec for more details. * Case 1 (valid): - ```bash - [annotations.com.hooks.ssh] - authorize_ssh_key = "/capstor/scratch/cscs//tests/edf/authorized_keys" - enabled = "true" - ``` + ```bash + [annotations.com.hooks.ssh] + authorize_ssh_key = "/capstor/scratch/cscs//tests/edf/authorized_keys" + enabled = "true" + ``` * Case 2 (valid): ```bash From e77dccc0ac8c940d508547b97315be807eaea27c Mon Sep 17 00:00:00 2001 From: Gwangmu Lee Date: Tue, 25 Feb 2025 13:18:01 +0100 Subject: [PATCH 16/34] Monospace-ize some words --- docs/tools/container-engine.md | 38 +++++++++++++++++----------------- 1 file changed, 19 insertions(+), 19 deletions(-) diff --git a/docs/tools/container-engine.md b/docs/tools/container-engine.md index 6f1edfb4..186c9778 100644 --- a/docs/tools/container-engine.md +++ b/docs/tools/container-engine.md @@ -325,16 +325,16 @@ As annotations are often used to control hooks, they have a deep nesting level. EDF files support setting annotations through the `annotations` table. This can be done in multiple ways in TOML: for example, both of the following usages are equivalent: * Case: nest levels in the TOML key. -```bash -[annotations] -com.hooks.ssh.enabled = "true" -``` + ```bash + [annotations] + com.hooks.ssh.enabled = "true" + ``` * Case: nest levels in the TOML table name. -```bash -[annotations.com.hooks.ssh] -enabled = "true" -``` + ```bash + [annotations.com.hooks.ssh] + enabled = "true" + ``` To avoid mistakes, notice a few key features of TOML: @@ -350,20 +350,20 @@ To avoid mistakes, notice a few key features of TOML: ``` * Case 2 (valid): - ```bash - [annotations] - com.hooks.ssh.authorize_ssh_key = "/capstor/scratch/cscs//tests/edf/authorized_keys" - com.hooks.ssh.enabled = "true" - ``` + ```bash + [annotations] + com.hooks.ssh.authorize_ssh_key = "/capstor/scratch/cscs//tests/edf/authorized_keys" + com.hooks.ssh.enabled = "true" + ``` * Case 3 (**invalid**): - ```bash - [annotations] - com.hooks.ssh.authorize_ssh_key = "/capstor/scratch/cscs//tests/edf/authorized_keys" + ```bash + [annotations] + com.hooks.ssh.authorize_ssh_key = "/capstor/scratch/cscs//tests/edf/authorized_keys" - [annotations.com.hooks.ssh] - enabled = "true" - ``` + [annotations.com.hooks.ssh] + enabled = "true" + ``` ## Accessing native resources From 79bb8b73123fcd545dab0bb30ade1133376cda07 Mon Sep 17 00:00:00 2001 From: Gwangmu Lee Date: Tue, 25 Feb 2025 13:33:53 +0100 Subject: [PATCH 17/34] Reference --- docs/tools/container-engine.md | 25 +++++++++++++------------ 1 file changed, 13 insertions(+), 12 deletions(-) diff --git a/docs/tools/container-engine.md b/docs/tools/container-engine.md index 186c9778..a2536145 100644 --- a/docs/tools/container-engine.md +++ b/docs/tools/container-engine.md @@ -625,34 +625,35 @@ com.hooks.nvidia_cuda_mps.enabled = "true" EDF files use the [TOML format](https://toml.io/en/). For details about the data types used by the different parameters, please refer to the [TOML spec webpage](https://toml.io/en/v1.0.0). -Parameter Description +In the following, the default value is none (i.e., the empty value of the corresponding type) if not specified. -base_environment - -Type: ARRAY or BASIC STRING - -Default: N/A - - + * *ARRAY or BASIC STRING* **base_environment** Ordered list of EDFs that this file inherits from. Parameters from listed environments are evaluated sequentially. Supports up to 10 levels of recursion. +
+Details + Parameters from the listed environments are evaluated sequentially, adding new entries or overwriting previous ones, before evaluating the parameters from the current EDF. In other words, the current EDF inherits the parameters from the EDFs listed in base_environment. When evaluating mounts or env parameters, values from downstream EDFs are appended to inherited values. The individual EDF entries in the array follow the same search rules as the arguments of the --environment CLI option for Slurm; they can be either file paths or filenames without extension if the file is located in the [EDF search path](#edf-search-path). This parameter can be a string if there is only one base environment. +
- - -Examples +
+Examples 1. Single environment inheritance: +```bash base_environment = "common_env" +``` 2. Multiple environment inheritance: +```bash base_environment = ["common_env", "ml_pytorch_env1"] - +``` +
image From 868c4652518a81c20857918a6f117090a25c0a27 Mon Sep 17 00:00:00 2001 From: Gwangmu Lee Date: Tue, 25 Feb 2025 13:35:00 +0100 Subject: [PATCH 18/34] Reference --- docs/tools/container-engine.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/tools/container-engine.md b/docs/tools/container-engine.md index a2536145..b14e1ffa 100644 --- a/docs/tools/container-engine.md +++ b/docs/tools/container-engine.md @@ -627,7 +627,7 @@ EDF files use the [TOML format](https://toml.io/en/). For details about the data In the following, the default value is none (i.e., the empty value of the corresponding type) if not specified. - * *ARRAY or BASIC STRING* **base_environment** + * **base_environment** Ordered list of EDFs that this file inherits from. Parameters from listed environments are evaluated sequentially. Supports up to 10 levels of recursion. From 302962be6176e93679ff8d26db143953dc96ca0a Mon Sep 17 00:00:00 2001 From: Gwangmu Lee Date: Tue, 25 Feb 2025 13:35:48 +0100 Subject: [PATCH 19/34] Reference --- docs/tools/container-engine.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/tools/container-engine.md b/docs/tools/container-engine.md index b14e1ffa..540e4d45 100644 --- a/docs/tools/container-engine.md +++ b/docs/tools/container-engine.md @@ -627,7 +627,7 @@ EDF files use the [TOML format](https://toml.io/en/). For details about the data In the following, the default value is none (i.e., the empty value of the corresponding type) if not specified. - * **base_environment** + * **base_environment** (Type: ARRAY or STRING) Ordered list of EDFs that this file inherits from. Parameters from listed environments are evaluated sequentially. Supports up to 10 levels of recursion. @@ -657,7 +657,7 @@ base_environment = ["common_env", "ml_pytorch_env1"] image -Type: BASIC STRING +Type: STRING Default: N/A @@ -699,7 +699,7 @@ image = "/path/to/image.squashfs" workdir -Type: BASIC STRING +Type: STRING Default: From cd0e9471ba28517f9f957314f3ab20ab570478f6 Mon Sep 17 00:00:00 2001 From: Gwangmu Lee Date: Tue, 25 Feb 2025 13:36:09 +0100 Subject: [PATCH 20/34] Reference --- docs/tools/container-engine.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/tools/container-engine.md b/docs/tools/container-engine.md index 540e4d45..46d2537d 100644 --- a/docs/tools/container-engine.md +++ b/docs/tools/container-engine.md @@ -627,7 +627,7 @@ EDF files use the [TOML format](https://toml.io/en/). For details about the data In the following, the default value is none (i.e., the empty value of the corresponding type) if not specified. - * **base_environment** (Type: ARRAY or STRING) + * **base_environment** (Type: ARRAY or STRING) Ordered list of EDFs that this file inherits from. Parameters from listed environments are evaluated sequentially. Supports up to 10 levels of recursion. From 672350288acd8ddbc3e8ea0b9e4f5484443c8c81 Mon Sep 17 00:00:00 2001 From: Gwangmu Lee Date: Tue, 25 Feb 2025 13:36:42 +0100 Subject: [PATCH 21/34] Reference --- docs/tools/container-engine.md | 36 +++++++++++++++++----------------- 1 file changed, 18 insertions(+), 18 deletions(-) diff --git a/docs/tools/container-engine.md b/docs/tools/container-engine.md index 46d2537d..54cf3b03 100644 --- a/docs/tools/container-engine.md +++ b/docs/tools/container-engine.md @@ -629,31 +629,31 @@ In the following, the default value is none (i.e., the empty value of the corres * **base_environment** (Type: ARRAY or STRING) -Ordered list of EDFs that this file inherits from. Parameters from listed environments are evaluated sequentially. Supports up to 10 levels of recursion. + Ordered list of EDFs that this file inherits from. Parameters from listed environments are evaluated sequentially. Supports up to 10 levels of recursion. -
-Details +
+ Details -Parameters from the listed environments are evaluated sequentially, adding new entries or overwriting previous ones, before evaluating the parameters from the current EDF. In other words, the current EDF inherits the parameters from the EDFs listed in base_environment. When evaluating mounts or env parameters, values from downstream EDFs are appended to inherited values. -The individual EDF entries in the array follow the same search rules as the arguments of the --environment CLI option for Slurm; they can be either file paths or filenames without extension if the file is located in the [EDF search path](#edf-search-path). -This parameter can be a string if there is only one base environment. -
+ Parameters from the listed environments are evaluated sequentially, adding new entries or overwriting previous ones, before evaluating the parameters from the current EDF. In other words, the current EDF inherits the parameters from the EDFs listed in base_environment. When evaluating mounts or env parameters, values from downstream EDFs are appended to inherited values. + The individual EDF entries in the array follow the same search rules as the arguments of the --environment CLI option for Slurm; they can be either file paths or filenames without extension if the file is located in the [EDF search path](#edf-search-path). + This parameter can be a string if there is only one base environment. +
-
-Examples +
+ Examples -1. Single environment inheritance: + 1. Single environment inheritance: -```bash -base_environment = "common_env" -``` + ```bash + base_environment = "common_env" + ``` -2. Multiple environment inheritance: + 2. Multiple environment inheritance: -```bash -base_environment = ["common_env", "ml_pytorch_env1"] -``` -
+ ```bash + base_environment = ["common_env", "ml_pytorch_env1"] + ``` +
image From 825e474cc9b17b3291ccda1682328d4830734f64 Mon Sep 17 00:00:00 2001 From: Gwangmu Lee Date: Tue, 25 Feb 2025 13:37:34 +0100 Subject: [PATCH 22/34] Reference --- docs/tools/container-engine.md | 18 ++++++++---------- 1 file changed, 8 insertions(+), 10 deletions(-) diff --git a/docs/tools/container-engine.md b/docs/tools/container-engine.md index 54cf3b03..78ff3fc5 100644 --- a/docs/tools/container-engine.md +++ b/docs/tools/container-engine.md @@ -642,17 +642,15 @@ In the following, the default value is none (i.e., the empty value of the corres
Examples - 1. Single environment inheritance: - - ```bash - base_environment = "common_env" - ``` - - 2. Multiple environment inheritance: + * Single environment inheritance: + ```bash + base_environment = "common_env" + ``` - ```bash - base_environment = ["common_env", "ml_pytorch_env1"] - ``` + * Multiple environment inheritance: + ```bash + base_environment = ["common_env", "ml_pytorch_env1"] + ```
image From 3ab95beb4403a4357cec846509f17057dbf7bdbe Mon Sep 17 00:00:00 2001 From: Gwangmu Lee Date: Tue, 25 Feb 2025 13:38:17 +0100 Subject: [PATCH 23/34] Reference --- docs/tools/container-engine.md | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-) diff --git a/docs/tools/container-engine.md b/docs/tools/container-engine.md index 78ff3fc5..3f6185c0 100644 --- a/docs/tools/container-engine.md +++ b/docs/tools/container-engine.md @@ -632,11 +632,13 @@ In the following, the default value is none (i.e., the empty value of the corres Ordered list of EDFs that this file inherits from. Parameters from listed environments are evaluated sequentially. Supports up to 10 levels of recursion.
- Details + Notes - Parameters from the listed environments are evaluated sequentially, adding new entries or overwriting previous ones, before evaluating the parameters from the current EDF. In other words, the current EDF inherits the parameters from the EDFs listed in base_environment. When evaluating mounts or env parameters, values from downstream EDFs are appended to inherited values. - The individual EDF entries in the array follow the same search rules as the arguments of the --environment CLI option for Slurm; they can be either file paths or filenames without extension if the file is located in the [EDF search path](#edf-search-path). - This parameter can be a string if there is only one base environment. + * Parameters from the listed environments are evaluated sequentially, adding new entries or overwriting previous ones, before evaluating the parameters from the current EDF. In other words, the current EDF inherits the parameters from the EDFs listed in base_environment. When evaluating mounts or env parameters, values from downstream EDFs are appended to inherited values. + + * The individual EDF entries in the array follow the same search rules as the arguments of the --environment CLI option for Slurm; they can be either file paths or filenames without extension if the file is located in the [EDF search path](#edf-search-path). + + * This parameter can be a string if there is only one base environment.
From 61a98318c8dbbf7f25acf27d0598b8c4b1997c19 Mon Sep 17 00:00:00 2001 From: Gwangmu Lee Date: Tue, 25 Feb 2025 13:52:09 +0100 Subject: [PATCH 24/34] Reference --- docs/tools/container-engine.md | 67 +++++++++++++++++----------------- 1 file changed, 34 insertions(+), 33 deletions(-) diff --git a/docs/tools/container-engine.md b/docs/tools/container-engine.md index 3f6185c0..1ae661a5 100644 --- a/docs/tools/container-engine.md +++ b/docs/tools/container-engine.md @@ -627,7 +627,7 @@ EDF files use the [TOML format](https://toml.io/en/). For details about the data In the following, the default value is none (i.e., the empty value of the corresponding type) if not specified. - * **base_environment** (Type: ARRAY or STRING) + * **base_environment** (Type: ARRAY or STRING) Ordered list of EDFs that this file inherits from. Parameters from listed environments are evaluated sequentially. Supports up to 10 levels of recursion. @@ -655,47 +655,48 @@ In the following, the default value is none (i.e., the empty value of the corres ```
-image - -Type: STRING - -Default: N/A - - - -The container image to use. Can reference a remote Docker/OCI registry or a local Squashfs file as a filesystem path. - -The full format for remote references is [USER@][REGISTRY#]IMAGE[:TAG]. -[REGISTRY#] : (optional) registry URL, followed by #. Default: Docker Hub. -IMAGE : image name. -[:TAG] : (optional) image tag name, preceded by :. -The registry user can also be specified in the $HOME/.config/enroot/.credentials file. -The default registry is Docker Hub (docker.io). + * **image** (Type: STRING) + The container image to use. Can reference a remote Docker/OCI registry or a local Squashfs file as a filesystem path. +
+ Notes -Examples - -1. Reference of Ubuntu image in the Docker Hub registry (default registry) - -image = "library/ubuntu:24.04" - -2. Explicit reference of Ubuntu image in the Docker Hub registry - -image = "docker.io#library/ubuntu:24.04" - -3. Reference to PyTorch image from NVIDIA Container Registry (nvcr.io) + * The full format for remote references is [USER@][REGISTRY#]IMAGE[:TAG]. + * [REGISTRY#] : (optional) registry URL, followed by #. Default: Docker Hub. + * IMAGE : image name. + * [:TAG] : (optional) image tag name, preceded by :. + * The registry user can also be specified in the $HOME/.config/enroot/.credentials file. +
-image = "nvcr.io#nvidia/pytorch:22.12-py3" +
+ Examples -5. Image from third-party quay.io registry + * Reference of Ubuntu image in the Docker Hub registry (default registry) + ```bash + image = "library/ubuntu:24.04" + ``` -image = "quay.io#madeeks/osu-mb:6.2-mpich4.1-ubuntu22.04-arm64" + * Explicit reference of Ubuntu image in the Docker Hub registry + ```bash + image = "docker.io#library/ubuntu:24.04" + ``` -4. Reference to a manually pulled image stored in parallel FS + * Reference to PyTorch image from NVIDIA Container Registry (nvcr.io) + ```bash + image = "nvcr.io#nvidia/pytorch:22.12-py3" + ``` -image = "/path/to/image.squashfs" + * Image from third-party quay.io registry + ```bash + image = "quay.io#madeeks/osu-mb:6.2-mpich4.1-ubuntu22.04-arm64" + ``` + * Reference to a manually pulled image stored in parallel FS + ```bash + image = "/path/to/image.squashfs" + ``` +
workdir From fc8b4e1f1e5b8ff594428e7393836587e3c2648b Mon Sep 17 00:00:00 2001 From: Gwangmu Lee Date: Tue, 25 Feb 2025 13:53:14 +0100 Subject: [PATCH 25/34] Reference --- docs/tools/container-engine.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/tools/container-engine.md b/docs/tools/container-engine.md index 1ae661a5..bacc9d9e 100644 --- a/docs/tools/container-engine.md +++ b/docs/tools/container-engine.md @@ -627,7 +627,7 @@ EDF files use the [TOML format](https://toml.io/en/). For details about the data In the following, the default value is none (i.e., the empty value of the corresponding type) if not specified. - * **base_environment** (Type: ARRAY or STRING) +### (ARRAY or STRING) base_environment Ordered list of EDFs that this file inherits from. Parameters from listed environments are evaluated sequentially. Supports up to 10 levels of recursion. From 3f2e2f5243707050ed803e43ab8a2aff1a022e1f Mon Sep 17 00:00:00 2001 From: Gwangmu Lee Date: Tue, 25 Feb 2025 13:53:53 +0100 Subject: [PATCH 26/34] Reference --- docs/tools/container-engine.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/docs/tools/container-engine.md b/docs/tools/container-engine.md index bacc9d9e..f3790486 100644 --- a/docs/tools/container-engine.md +++ b/docs/tools/container-engine.md @@ -627,7 +627,9 @@ EDF files use the [TOML format](https://toml.io/en/). For details about the data In the following, the default value is none (i.e., the empty value of the corresponding type) if not specified. -### (ARRAY or STRING) base_environment +### base_environment + + Type: ARRAY or STRING Ordered list of EDFs that this file inherits from. Parameters from listed environments are evaluated sequentially. Supports up to 10 levels of recursion. From e80b5674c0aecf1aa842b05e0f5bf729ad363840 Mon Sep 17 00:00:00 2001 From: Gwangmu Lee Date: Tue, 25 Feb 2025 13:54:18 +0100 Subject: [PATCH 27/34] Reference --- docs/tools/container-engine.md | 40 +++++++++++++++++----------------- 1 file changed, 20 insertions(+), 20 deletions(-) diff --git a/docs/tools/container-engine.md b/docs/tools/container-engine.md index f3790486..1dfbeff8 100644 --- a/docs/tools/container-engine.md +++ b/docs/tools/container-engine.md @@ -629,33 +629,33 @@ In the following, the default value is none (i.e., the empty value of the corres ### base_environment - Type: ARRAY or STRING +Type: ARRAY or STRING - Ordered list of EDFs that this file inherits from. Parameters from listed environments are evaluated sequentially. Supports up to 10 levels of recursion. +Ordered list of EDFs that this file inherits from. Parameters from listed environments are evaluated sequentially. Supports up to 10 levels of recursion. -
- Notes +
+Notes - * Parameters from the listed environments are evaluated sequentially, adding new entries or overwriting previous ones, before evaluating the parameters from the current EDF. In other words, the current EDF inherits the parameters from the EDFs listed in base_environment. When evaluating mounts or env parameters, values from downstream EDFs are appended to inherited values. + * Parameters from the listed environments are evaluated sequentially, adding new entries or overwriting previous ones, before evaluating the parameters from the current EDF. In other words, the current EDF inherits the parameters from the EDFs listed in base_environment. When evaluating mounts or env parameters, values from downstream EDFs are appended to inherited values. - * The individual EDF entries in the array follow the same search rules as the arguments of the --environment CLI option for Slurm; they can be either file paths or filenames without extension if the file is located in the [EDF search path](#edf-search-path). - - * This parameter can be a string if there is only one base environment. -
+ * The individual EDF entries in the array follow the same search rules as the arguments of the --environment CLI option for Slurm; they can be either file paths or filenames without extension if the file is located in the [EDF search path](#edf-search-path). -
- Examples + * This parameter can be a string if there is only one base environment. +
- * Single environment inheritance: - ```bash - base_environment = "common_env" - ``` +
+Examples - * Multiple environment inheritance: - ```bash - base_environment = ["common_env", "ml_pytorch_env1"] - ``` -
+ * Single environment inheritance: + ```bash + base_environment = "common_env" + ``` + + * Multiple environment inheritance: + ```bash + base_environment = ["common_env", "ml_pytorch_env1"] + ``` +
* **image** (Type: STRING) From 94dcd6d0bd650b12cff3ed0664eeda6a84b8ba1b Mon Sep 17 00:00:00 2001 From: Gwangmu Lee Date: Tue, 25 Feb 2025 13:55:06 +0100 Subject: [PATCH 28/34] Reference --- docs/tools/container-engine.md | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/docs/tools/container-engine.md b/docs/tools/container-engine.md index 1dfbeff8..b6d27c6d 100644 --- a/docs/tools/container-engine.md +++ b/docs/tools/container-engine.md @@ -627,9 +627,7 @@ EDF files use the [TOML format](https://toml.io/en/). For details about the data In the following, the default value is none (i.e., the empty value of the corresponding type) if not specified. -### base_environment - -Type: ARRAY or STRING +### base_environment (ARRAY or STRING) Ordered list of EDFs that this file inherits from. Parameters from listed environments are evaluated sequentially. Supports up to 10 levels of recursion. From 9c1aaae9cf70d81b10b5ca8860e6286e394f0860 Mon Sep 17 00:00:00 2001 From: Gwangmu Lee Date: Tue, 25 Feb 2025 13:56:56 +0100 Subject: [PATCH 29/34] Reference --- docs/tools/container-engine.md | 21 +++++++++++---------- 1 file changed, 11 insertions(+), 10 deletions(-) diff --git a/docs/tools/container-engine.md b/docs/tools/container-engine.md index b6d27c6d..268d4eef 100644 --- a/docs/tools/container-engine.md +++ b/docs/tools/container-engine.md @@ -655,19 +655,20 @@ Ordered list of EDFs that this file inherits from. Parameters from listed enviro ``` - * **image** (Type: STRING) +### image (STRING) - The container image to use. Can reference a remote Docker/OCI registry or a local Squashfs file as a filesystem path. +The container image to use. Can reference a remote Docker/OCI registry or a local Squashfs file as a filesystem path. -
- Notes +
+Notes - * The full format for remote references is [USER@][REGISTRY#]IMAGE[:TAG]. - * [REGISTRY#] : (optional) registry URL, followed by #. Default: Docker Hub. - * IMAGE : image name. - * [:TAG] : (optional) image tag name, preceded by :. - * The registry user can also be specified in the $HOME/.config/enroot/.credentials file. -
+The full format for remote references is [USER@][REGISTRY#]IMAGE[:TAG]. + * [REGISTRY#] : (optional) registry URL, followed by #. Default: Docker Hub. + * IMAGE : image name. + * [:TAG] : (optional) image tag name, preceded by :. + +The registry user can also be specified in the $HOME/.config/enroot/.credentials file. +
Examples From b48b10720dd231e18ab2a8ce6259953bc26b0f9c Mon Sep 17 00:00:00 2001 From: Gwangmu Lee Date: Tue, 25 Feb 2025 13:57:47 +0100 Subject: [PATCH 30/34] Reference --- docs/tools/container-engine.md | 57 +++++++++++++++++----------------- 1 file changed, 28 insertions(+), 29 deletions(-) diff --git a/docs/tools/container-engine.md b/docs/tools/container-engine.md index 268d4eef..eabbea38 100644 --- a/docs/tools/container-engine.md +++ b/docs/tools/container-engine.md @@ -662,42 +662,41 @@ The container image to use. Can reference a remote Docker/OCI registry or a loca
Notes -The full format for remote references is [USER@][REGISTRY#]IMAGE[:TAG]. - * [REGISTRY#] : (optional) registry URL, followed by #. Default: Docker Hub. - * IMAGE : image name. - * [:TAG] : (optional) image tag name, preceded by :. - -The registry user can also be specified in the $HOME/.config/enroot/.credentials file. + * The full format for remote references is [USER@][REGISTRY#]IMAGE[:TAG]. + * [REGISTRY#] : (optional) registry URL, followed by #. Default: Docker Hub. + * IMAGE : image name. + * [:TAG] : (optional) image tag name, preceded by :. + * The registry user can also be specified in the $HOME/.config/enroot/.credentials file.
-
- Examples +
+Examples - * Reference of Ubuntu image in the Docker Hub registry (default registry) - ```bash - image = "library/ubuntu:24.04" - ``` + * Reference of Ubuntu image in the Docker Hub registry (default registry) + ```bash + image = "library/ubuntu:24.04" + ``` - * Explicit reference of Ubuntu image in the Docker Hub registry - ```bash - image = "docker.io#library/ubuntu:24.04" - ``` + * Explicit reference of Ubuntu image in the Docker Hub registry + ```bash + image = "docker.io#library/ubuntu:24.04" + ``` - * Reference to PyTorch image from NVIDIA Container Registry (nvcr.io) - ```bash - image = "nvcr.io#nvidia/pytorch:22.12-py3" - ``` + * Reference to PyTorch image from NVIDIA Container Registry (nvcr.io) + ```bash + image = "nvcr.io#nvidia/pytorch:22.12-py3" + ``` - * Image from third-party quay.io registry - ```bash - image = "quay.io#madeeks/osu-mb:6.2-mpich4.1-ubuntu22.04-arm64" - ``` + * Image from third-party quay.io registry + ```bash + image = "quay.io#madeeks/osu-mb:6.2-mpich4.1-ubuntu22.04-arm64" + ``` - * Reference to a manually pulled image stored in parallel FS - ```bash - image = "/path/to/image.squashfs" - ``` -
+ * Reference to a manually pulled image stored in parallel FS + ```bash + image = "/path/to/image.squashfs" + ``` +
workdir From c60b0b2be6521564d09dd8a39de633faa7c1a254 Mon Sep 17 00:00:00 2001 From: Gwangmu Lee Date: Tue, 25 Feb 2025 14:09:36 +0100 Subject: [PATCH 31/34] Reference --- docs/tools/container-engine.md | 248 +++++++++++++-------------------- 1 file changed, 100 insertions(+), 148 deletions(-) diff --git a/docs/tools/container-engine.md b/docs/tools/container-engine.md index eabbea38..ce2643ff 100644 --- a/docs/tools/container-engine.md +++ b/docs/tools/container-engine.md @@ -698,191 +698,143 @@ The container image to use. Can reference a remote Docker/OCI registry or a loca ```
-workdir +### workdir (STRING) -Type: STRING +Initial working directory when the container starts. Default: inherited from image. -Default: - -Inherited from image - - - -Initial working directory when the container starts. - - - - -Examples - -1. Workdir pointing to a user defined project path  - -workdir = "/home/user/projects" - -2. Workdir pointing to the /tmp directory - -workdir = "/tmp" - - -entrypoint - -Type: BOOL - -Default: true +
+Example - + * Workdir pointing to a user defined project path  + ```bash + workdir = "/home/user/projects" + ``` -If true, run the entrypoint from the container image + * Workdir pointing to the /tmp directory + ```bash + workdir = "/tmp" + ``` +
+### entrypoint (BOOL) +If true, run the entrypoint from the container image. Default: true. -Examples +
+Example +```bash entrypoint = false +``` +
+### writable (BOOL) +If false, the container filesystem is read-only. Default: false. -writable - -Type: BOOL - -Default: false - - - -If false, the container filesystem is read-only - - - - -Examples +
+Example +```bash writable = true +``` +
- - -mounts - -Type: ARRAY - -Default: N/A - - +### mounts (ARRAY) List of bind mounts in the format SOURCE:DESTINATION[:FLAGS]. Flags are optional and can include ro, private, etc. -Mount flags are separated with a plus symbol, for example: ro+private. -Optional flags from docker format or OCI (need reference) - - - - - - - -Examples - -Literal fixed mount map -mounts = ["/capstor/scratch/cscs/amadonna:/capstor/scratch/cscs/amadonna"] - -2. Mapping path with env variable expansion - -mounts = ["/capstor/scratch/cscs/${USER}:/capstor/scratch/cscs/${USER}"] - -3. Mounting the scratch filesystem using a host environment variable - -mounts = ["${SCRATCH}:/scratch"] - - - - -env - -Type: TABLE - -Default: - -Inherited from - -host and - -image - - - -Environment variables to set in the container. Null-string values will unset the variable. - -By default, containers inherit environment variables from the container image and the host environment, with variables from the image taking precedence. -The env table can be used to further customize the container environment by setting, modifying, or unsetting variables. -Values of the table entries must be strings. If an entry has a null value, the variable corresponding to the entry key is unset in the container. - - - -Examples - -1. Basic env block - -[env] -MY_RUN = "production", -DEBUG = "false" -2. Use of environment variable expansion -[env] -MY_NODE = "${VAR_FROM_HOST}", -PATH = "${PATH}:/custom/bin", -DEBUG = "true" - - - - - - - - -annotations - -Type: TABLE +
+Notes -Default: N/A + * Mount flags are separated with a plus symbol, for example: ro+private. + * Optional flags from docker format or OCI (need reference) +
- +
+Example -OCI-like annotations for the container. + * Literal fixed mount map + ```bash + mounts = ["/capstor/scratch/cscs/amadonna:/capstor/scratch/cscs/amadonna"] + ``` -For more details, refer to the Annotations section. + * Mapping path with env variable expansion + ```bash + mounts = ["/capstor/scratch/cscs/${USER}:/capstor/scratch/cscs/${USER}"] + ``` -  + * Mounting the scratch filesystem using a host environment variable + ```bash + mounts = ["${SCRATCH}:/scratch"] + ``` +
+### env (TABLE) +Environment variables to set in the container. Null-string values will unset the variable. Default: inherited from the host and the image. -Examples +
+Notes -1. Disabling the CXI hook + * By default, containers inherit environment variables from the container image and the host environment, with variables from the image taking precedence. + * The env table can be used to further customize the container environment by setting, modifying, or unsetting variables. + * Values of the table entries must be strings. If an entry has a null value, the variable corresponding to the entry key is unset in the container. +
-[annotations] -com.hooks.cxi.enabled = "false" +
+Example -2. Control of SSH hook parameters via annotation and variable expansion + * Basic env block + ```bash + [env] + MY_RUN = "production", + DEBUG = "false" + ``` -[annotations.com.hooks.ssh] -authorize_ssh_key = "/capstor/scratch/cscs/${USER}/tests/edf/authorized_keys" -enabled = "true" + * Use of environment variable expansion + ```bash + [env] + MY_NODE = "${VAR_FROM_HOST}", + PATH = "${PATH}:/custom/bin", + DEBUG = "true" + ``` +
-3. Alternative example for usage of annotation with fixed path -[annotations] -com.hooks.ssh.authorize_ssh_key = "/path/to/authorized_keys" -com.hooks.ssh.enabled = "true" +### annotations (TABLE) +OCI-like annotations for the container. For more details, refer to the Annotations section. +
+Example + * Disabling the CXI hook + ```bash + [annotations] + com.hooks.cxi.enabled = "false" + ``` + * Control of SSH hook parameters via annotation and variable expansion + ```bash + [annotations.com.hooks.ssh] + authorize_ssh_key = "/capstor/scratch/cscs/${USER}/tests/edf/authorized_keys" + enabled = "true" + ``` + * Alternative example for usage of annotation with fixed path + ```bash + [annotations] + com.hooks.ssh.authorize_ssh_key = "/path/to/authorized_keys" + com.hooks.ssh.enabled = "true" + ``` +
-Environment variable expansion and relative paths expansion are only available on the Bristen vCluster as technical preview. +> **INFO**: Environment variable expansion and relative paths expansion are only available on the Bristen vCluster as technical preview. ### Environment Variable Expansion @@ -895,9 +847,9 @@ Environment variable expansion allows for dynamic substitution of environment va * *Limitations* * Variables defined within the [env] EDF table cannot reference other entries from [env] tables in the same or other EDF files (e.g. the ones entered as base environments) . Therefore, only environment variables from the host or image can be referenced. * *Environment Variable Resolution Order*. The environment variables are resolved based on the following order: - 1. TOML env: Variable values as defined in EDF’s env. - 2. Container Image: Variables defined in the container image's environment take precedence. - 3. Host Environment: Environment variables defined in the host system. + * TOML env: Variable values as defined in EDF’s env. + * Container Image: Variables defined in the container image's environment take precedence. + * Host Environment: Environment variables defined in the host system. ### Relative paths expansion @@ -911,7 +863,7 @@ Alpine Linux is incompatible with some hooks, causing errors when used with Slur ```bash [][@-ln001 ~]$ cat alpine.toml -image = "alpine:3.19" +image = "alpine: *19" [][@-ln001 ~]$ srun -lN1 --environment=alpine.toml echo "abc" 0: slurmstepd: error: pyxis: container start failed with error code: 1 0: slurmstepd: error: pyxis: printing enroot log file: @@ -926,7 +878,7 @@ This is because some hooks (e.g., Slurm and CXI hooks) leverage `ldconfig` (from ```bash [][@-ln001 ~]$ cat alpine_workaround.toml -image = "alpine:3.19" +image = "alpine: *19" [annotations] com.hooks.slurm.enabled = "false" com.hooks.cxi.enabled = "false" From 49ebf1b9ddf947ebda96c992dd8e747aaed1d622 Mon Sep 17 00:00:00 2001 From: Gwangmu Lee Date: Tue, 25 Feb 2025 14:10:49 +0100 Subject: [PATCH 32/34] Reference --- docs/tools/container-engine.md | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/docs/tools/container-engine.md b/docs/tools/container-engine.md index ce2643ff..3b0f2ecc 100644 --- a/docs/tools/container-engine.md +++ b/docs/tools/container-engine.md @@ -655,7 +655,7 @@ Ordered list of EDFs that this file inherits from. Parameters from listed enviro ``` -### image (STRING) +### (STRING) image The container image to use. Can reference a remote Docker/OCI registry or a local Squashfs file as a filesystem path. @@ -698,7 +698,7 @@ The container image to use. Can reference a remote Docker/OCI registry or a loca ``` -### workdir (STRING) +### (STRING) workdir Initial working directory when the container starts. Default: inherited from image. @@ -717,7 +717,7 @@ Initial working directory when the container starts. Default: inherited from ima -### entrypoint (BOOL) +### (BOOL) entrypoint If true, run the entrypoint from the container image. Default: true. @@ -730,7 +730,7 @@ entrypoint = false -### writable (BOOL) +### (BOOL) writable If false, the container filesystem is read-only. Default: false. @@ -743,7 +743,7 @@ writable = true -### mounts (ARRAY) +### (ARRAY) mounts List of bind mounts in the format SOURCE:DESTINATION[:FLAGS]. Flags are optional and can include ro, private, etc. @@ -774,7 +774,7 @@ List of bind mounts in the format SOURCE:DESTINATION[:FLAGS]. Flags are optional -### env (TABLE) +### (TABLE) env Environment variables to set in the container. Null-string values will unset the variable. Default: inherited from the host and the image. @@ -806,7 +806,7 @@ Environment variables to set in the container. Null-string values will unset the -### annotations (TABLE) +### (TABLE) annotations OCI-like annotations for the container. For more details, refer to the Annotations section. From 4977dd64ff36a739abb672721eeece55606bb872 Mon Sep 17 00:00:00 2001 From: Gwangmu Lee Date: Tue, 25 Feb 2025 14:11:14 +0100 Subject: [PATCH 33/34] Reference --- docs/tools/container-engine.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/tools/container-engine.md b/docs/tools/container-engine.md index 3b0f2ecc..95627804 100644 --- a/docs/tools/container-engine.md +++ b/docs/tools/container-engine.md @@ -627,7 +627,7 @@ EDF files use the [TOML format](https://toml.io/en/). For details about the data In the following, the default value is none (i.e., the empty value of the corresponding type) if not specified. -### base_environment (ARRAY or STRING) +### (ARRAY or STRING) base_environment Ordered list of EDFs that this file inherits from. Parameters from listed environments are evaluated sequentially. Supports up to 10 levels of recursion. From 028ab385ddb94795cbc13fa549533f9c6497fc54 Mon Sep 17 00:00:00 2001 From: Gwangmu Lee Date: Tue, 25 Feb 2025 14:16:15 +0100 Subject: [PATCH 34/34] Reference --- docs/tools/container-engine.md | 28 ++++++++++++++-------------- 1 file changed, 14 insertions(+), 14 deletions(-) diff --git a/docs/tools/container-engine.md b/docs/tools/container-engine.md index 95627804..75dd162c 100644 --- a/docs/tools/container-engine.md +++ b/docs/tools/container-engine.md @@ -316,7 +316,7 @@ machine registry.ethz.ch login password machine gitlab.ethz.ch login password ``` -## Annotations  +## Annotations  Annotations define arbitrary metadata for containers in the form of key-value pairs. Within the EDF, annotations are designed to be similar in appearance and behavior to those defined by the [OCI Runtime Specification](https://github.com/opencontainers/runtime-spec/blob/main/config.md#annotations). Annotation keys usually express a hierarchical namespace structure, with domains separated by "." (full stop) characters. @@ -634,9 +634,9 @@ Ordered list of EDFs that this file inherits from. Parameters from listed enviro
Notes - * Parameters from the listed environments are evaluated sequentially, adding new entries or overwriting previous ones, before evaluating the parameters from the current EDF. In other words, the current EDF inherits the parameters from the EDFs listed in base_environment. When evaluating mounts or env parameters, values from downstream EDFs are appended to inherited values. + * Parameters from the listed environments are evaluated sequentially, adding new entries or overwriting previous ones, before evaluating the parameters from the current EDF. In other words, the current EDF inherits the parameters from the EDFs listed in `base_environment`. When evaluating `mounts` or `env` parameters, values from downstream EDFs are appended to inherited values. - * The individual EDF entries in the array follow the same search rules as the arguments of the --environment CLI option for Slurm; they can be either file paths or filenames without extension if the file is located in the [EDF search path](#edf-search-path). + * The individual EDF entries in the array follow the same search rules as the arguments of the `--environment` CLI option for Slurm; they can be either file paths or filenames without extension if the file is located in the [EDF search path](#edf-search-path). * This parameter can be a string if there is only one base environment.
@@ -662,11 +662,11 @@ The container image to use. Can reference a remote Docker/OCI registry or a loca
Notes - * The full format for remote references is [USER@][REGISTRY#]IMAGE[:TAG]. - * [REGISTRY#] : (optional) registry URL, followed by #. Default: Docker Hub. - * IMAGE : image name. - * [:TAG] : (optional) image tag name, preceded by :. - * The registry user can also be specified in the $HOME/.config/enroot/.credentials file. + * The full format for remote references is `[USER@][REGISTRY#]IMAGE[:TAG]`. + * `[REGISTRY#]`: (optional) registry URL, followed by #. Default: Docker Hub. + * `IMAGE`: image name. + * `[:TAG]`: (optional) image tag name, preceded by :. + * The registry user can also be specified in the `$HOME/.config/enroot/.credentials` file.
@@ -710,7 +710,7 @@ Initial working directory when the container starts. Default: inherited from ima workdir = "/home/user/projects" ``` - * Workdir pointing to the /tmp directory + * Workdir pointing to the `/tmp` directory ```bash workdir = "/tmp" ``` @@ -745,12 +745,12 @@ writable = true ### (ARRAY) mounts -List of bind mounts in the format SOURCE:DESTINATION[:FLAGS]. Flags are optional and can include ro, private, etc. +List of bind mounts in the format `SOURCE:DESTINATION[:FLAGS]`. Flags are optional and can include `ro`, `private`, etc.
Notes - * Mount flags are separated with a plus symbol, for example: ro+private. + * Mount flags are separated with a plus symbol, for example: `ro+private`. * Optional flags from docker format or OCI (need reference)
@@ -762,7 +762,7 @@ List of bind mounts in the format SOURCE:DESTINATION[:FLAGS]. Flags are optional mounts = ["/capstor/scratch/cscs/amadonna:/capstor/scratch/cscs/amadonna"] ``` - * Mapping path with env variable expansion + * Mapping path with `env` variable expansion ```bash mounts = ["/capstor/scratch/cscs/${USER}:/capstor/scratch/cscs/${USER}"] ``` @@ -789,7 +789,7 @@ Environment variables to set in the container. Null-string values will unset the
Example - * Basic env block + * Basic `env` block ```bash [env] MY_RUN = "production", @@ -808,7 +808,7 @@ Environment variables to set in the container. Null-string values will unset the ### (TABLE) annotations -OCI-like annotations for the container. For more details, refer to the Annotations section. +OCI-like annotations for the container. For more details, refer to the [Annotations](#annotations) section.
Example