Merge branch 'main' into update-eiger

Author: bcumming

docs/clusters/bristen.md

@@ -12,7 +12,7 @@ Bristen consists of 32 A100 nodes [NVIDIA A100 nodes][ref-alps-a100-node]. The n
 |-----------|--------| ----------------- | ---------- |
 | [a100][ref-alps-a100-node] | 32 | 32 | 128 |
 
-Nodes are in the [`normal` slurm partition][ref-slurm-partition-normal].
+Nodes are in the [`normal` Slurm partition][ref-slurm-partition-normal].
 
 ### Storage and file systems
 
@@ -48,7 +48,7 @@ Users are encouraged to use containers on Bristen.
 
 Bristen uses [Slurm][ref-slurm] as the workload manager, which is used to launch and monitor distributed workloads, such as training runs.
 
-There is currently a single slurm partition on the system:
+There is currently a single Slurm partition on the system:
 
 * the `normal` partition is for all production workloads.
     + nodes in this partition are not shared.

docs/clusters/clariden.md

@@ -14,7 +14,7 @@ The number of nodes can change when nodes are added or removed from other cluste
 |-----------|--------| ----------------- | ---------- |
 | [gh200][ref-alps-gh200-node] | 1,200 | 4,800 | 4,800 |
 
-Most nodes are in the [`normal` slurm partition][ref-slurm-partition-normal], while a few nodes are in the [`debug` partition][ref-slurm-partition-debug].
+Most nodes are in the [`normal` Slurm partition][ref-slurm-partition-normal], while a few nodes are in the [`debug` partition][ref-slurm-partition-debug].
 
 ### Storage and file systems
 
@@ -71,7 +71,7 @@ Alternatively, [uenv][ref-uenv] are also available on Clariden. Currently deploy
 
 Clariden uses [Slurm][ref-slurm] as the workload manager, which is used to launch and monitor distributed workloads, such as training runs.
 
-There are two slurm partitions on the system:
+There are two Slurm partitions on the system:
 
 * the `normal` partition is for all production workloads.
 * the `debug` partition can be used to access a small allocation for up to 30 minutes for debugging and testing purposes.

docs/running/jobreport.md

@@ -56,7 +56,7 @@ The report is divided into two parts: a general summary and GPU specific values.
 | Field | Description |
 | ----- | ----------- |
 | Job Id | The Slurm job id |
-| Step Id | The slurm step id. A job step in Slurm is a subdivision of a job started with srun |
+| Step Id | The Slurm step id. A job step in Slurm is a subdivision of a job started with srun |
 | User | The user account that submitted the job |
 | Slurm Account | The project account that will be billed |
 | Start Time, End Time, Elapsed Time | The time the job started and ended, and how long it ran |
@@ -77,7 +77,7 @@ The report is divided into two parts: a general summary and GPU specific values.
 | SM Utilization % | The percentage of the process's lifetime during which Streaming Multiprocessors (SM) were executing a kernel |
 | Memory Utilization % | The percentage of process's lifetime during which global (device) memory was being read or written |
 
-## Example with slurm: srun
+## Example with Slurm: srun
 
 The simplest example to test `jobreport` is to run it with the sleep command.
 It is important to separate `jobreport` (and its options) and your command  with `--`.
@@ -155,7 +155,7 @@ GPU Specific Values
     4. Uncheck "Set locale environment variables on startup"
     5. Quit and reopen the terminal and try again. This should fix the issue.
 
-## Example with slurm: batch script
+## Example with Slurm: batch script
 
 The `jobreport` command can be used in a batch script
 The report printing, too, can be included in the script and does not need the `srun` command.

docs/running/slurm.md

@@ -1,8 +1,31 @@
 [](){#ref-slurm}
-# SLURM
+# Slurm
 
-CSCS uses the [SLURM](https://slurm.schedmd.com/documentation.html) as its workload manager to efficiently schedule and manage jobs on Alps vClusters.
-SLURM is an open-source, highly scalable job scheduler that allocates computing resources, queues user jobs, and optimizes workload distribution across the cluster. It supports advanced scheduling policies, job dependencies, resource reservations, and accounting, making it well-suited for high-performance computing environments.
+CSCS uses the [Slurm](https://slurm.schedmd.com/documentation.html) workload manager to efficiently schedule and manage jobs on Alps vClusters.
+Slurm is an open-source, highly scalable job scheduler that allocates computing resources, queues user jobs, and optimizes workload distribution across the cluster.
+It supports advanced scheduling policies, job dependencies, resource reservations, and accounting, making it well-suited for high-performance computing environments.
+
+Refer to the [Quick Start User Guide](https://slurm.schedmd.com/quickstart.html) for commonly used terminology and commands.
+
+<div class="grid cards" markdown>
+
+-   :fontawesome-solid-mountain-sun: __Configuring jobs__
+
+    Specific guidance for configuring Slurm jobs on different node types.
+
+    [:octicons-arrow-right-24: GH200 nodes (Daint, Clariden, Santis)][ref-slurm-gh200]
+
+    [:octicons-arrow-right-24: AMD CPU-only nodes (Eiger)][ref-slurm-amdcpu]
+
+-   :fontawesome-solid-mountain-sun: __Node sharing__
+
+    Guides on how to effectively use all resouces on nodes by running more than one job per node.
+
+    [:octicons-arrow-right-24: Node sharing][ref-slurm-sharing]
+
+    [:octicons-arrow-right-24: Multiple MPI jobs per node][ref-slurm-exclusive]
+
+</div>
 
 ## Accounts and resources
 
@@ -58,9 +81,9 @@ Additionally, short-duration jobs may be selected for backfilling — a process
 [](){#ref-slurm-partitions}
 ## Partitions
 
-At CSCS, SLURM is configured to accommodate the diverse range of node types available in our HPC clusters.
+At CSCS, Slurm is configured to accommodate the diverse range of node types available in our HPC clusters.
 These nodes vary in architecture, including CPU-only nodes and nodes equipped with different types of GPUs.
-Because of this heterogeneity, SLURM must be tailored to ensure efficient resource allocation, job scheduling, and workload management specific to each node type.
+Because of this heterogeneity, Slurm must be tailored to ensure efficient resource allocation, job scheduling, and workload management specific to each node type.
 
 Each type of node has different resource constraints and capabilities, which Slurm takes into account when scheduling jobs.
 For example, CPU-only nodes may have configurations optimized for multi-threaded CPU workloads, while GPU nodes require additional parameters to allocate GPU resources efficiently.
@@ -80,13 +103,15 @@ Slurm ensures that user jobs request and receive the appropriate resources while
 
 [](){#ref-slurm-partition-debug}
 ### Debug partition
-The SLURM `debug` partition is useful for quick turnaround workflows. The partition has a short maximum time (timelimit can be seen with `sinfo -p debug`), and a low number of maximum nodes (the `MaxNodes` can be seen with `scontrol show partition=debug`).
+The Slurm `debug` partition is useful for quick turnaround workflows. The partition has a short maximum time (timelimit can be seen with `sinfo -p debug`), and a low number of maximum nodes (the `MaxNodes` can be seen with `scontrol show partition=debug`).
 
 [](){#ref-slurm-partition-normal}
 ### Normal partition
-This is the default partition, and will be used when you do not explicitly set a partition. This is the correct choice for standard jobs. The maximum time is usually set to 24 hours (`sinfo -p normal` for timelimit), and the maximum nodes can be as much as nodes are available.
+This is the default partition, and will be used when you do not explicitly set a partition.
+This is the correct choice for standard jobs. The maximum time is usually set to 24 hours (`sinfo -p normal` for timelimit), and the maximum nodes can be as much as nodes are available.
 
-The following sections will provide detailed guidance on how to use SLURM to request and manage CPU cores, memory, and GPUs in jobs. These instructions will help users optimize their workload execution and ensure efficient use of CSCS computing resources.
+The following sections will provide detailed guidance on how to use Slurm to request and manage CPU cores, memory, and GPUs in jobs.
+These instructions will help users optimize their workload execution and ensure efficient use of CSCS computing resources.
 
 ## Affinity
 
@@ -215,13 +240,13 @@ The build generates the following executables:
 [](){#ref-slurm-gh200}
 ## NVIDIA GH200 GPU Nodes
 
-The [GH200 nodes on Alps][ref-alps-gh200-node] have four GPUs per node, and SLURM job submissions must be configured appropriately to best make use of the resources.
+The [GH200 nodes on Alps][ref-alps-gh200-node] have four GPUs per node, and Slurm job submissions must be configured appropriately to best make use of the resources.
 Applications that can saturate the GPUs with a single process per GPU should generally prefer this mode.
-[Configuring SLURM jobs to use a single GPU per rank][ref-slurm-gh200-single-rank-per-gpu] is also the most straightforward setup.
+[Configuring Slurm jobs to use a single GPU per rank][ref-slurm-gh200-single-rank-per-gpu] is also the most straightforward setup.
 Some applications perform badly with a single rank per GPU, and require use of [NVIDIA's Multi-Process Service (MPS)] to oversubscribe GPUs with multiple ranks per GPU.
 
-The best SLURM configuration is application- and workload-specific, so it is worth testing which works best in your particular case.
-See [Scientific Applications][ref-software-sciapps] for information about recommended application-specific SLURM configurations.
+The best Slurm configuration is application- and workload-specific, so it is worth testing which works best in your particular case.
+See [Scientific Applications][ref-software-sciapps] for information about recommended application-specific Slurm configurations.
 
 !!! warning
     The GH200 nodes have their GPUs configured in ["default" compute mode](https://docs.nvidia.com/deploy/mps/index.html#gpu-compute-modes).
@@ -232,12 +257,12 @@ See [Scientific Applications][ref-software-sciapps] for information about recomm
     Some applications benefit from using multiple ranks per GPU. However, [MPS should be used][ref-slurm-gh200-multi-rank-per-gpu] in these cases.
 
     If you are unsure about which GPU is being used for a particular rank, print the `CUDA_VISIBLE_DEVICES` variable, along with e.g. `SLURM_LOCALID`, `SLURM_PROCID`, and `SLURM_NODEID` variables, in your job script.
-    If the variable is unset or empty all GPUs are visible to the rank and the rank will in most cases only use the first GPU. 
+    If the variable is unset or empty all GPUs are visible to the rank and the rank will in most cases only use the first GPU.
 
 [](){#ref-slurm-gh200-single-rank-per-gpu}
 ### One rank per GPU
 
-Configuring SLURM to use one GH200 GPU per rank is easiest done using the `--ntasks-per-node=4` and `--gpus-per-task=1` SLURM flags.
+Configuring Slurm to use one GH200 GPU per rank is easiest done using the `--ntasks-per-node=4` and `--gpus-per-task=1` Slurm flags.
 For advanced users, using `--gpus-per-task` is equivalent to setting `CUDA_VISIBLE_DEVICES` to `SLURM_LOCALID`, assuming the job is using four ranks per node.
 The examples below launch jobs on two nodes with four ranks per node using `sbatch` and `srun`:
 
@@ -257,7 +282,7 @@ Omitting the `--gpus-per-task` results in `CUDA_VISIBLE_DEVICES` being unset, wh
 ### Multiple ranks per GPU
 
 Using multiple ranks per GPU can improve performance e.g. of applications that don't generate enough work for a GPU using a single rank, or ones that scale badly to all 72 cores of the Grace CPU.
-In these cases SLURM jobs must be configured to assign multiple ranks to a single GPU.
+In these cases Slurm jobs must be configured to assign multiple ranks to a single GPU.
 This is best done using [NVIDIA's Multi-Process Service (MPS)].
 To use MPS, launch your application using the following wrapper script, which will start MPS on one rank per node and assign GPUs to ranks according to the CPU mask of a rank, ensuring the closest GPU is used:
 
@@ -519,3 +544,117 @@ rank   3 @ nid001512
 !!! warning
     The `OMP_*` environment variables only affect thread affinity of applications that use OpenMP for thread-level parallelism.
     Other threading runtimes will be configured differently, and the `affinity.mpi` tool will only be able to show the set of cores assigned to the rank.
+
+[](){#ref-slurm-over-subscription}
+## Node over-subscription
+
+The nodes on Alps provide a lot of resources, particularly the GPU nodes that have 4 GPUs.
+For workflows and use cases with tasks that require only a subset of these resources, for example a simulation that only needs one GPU, allocating a whole node to run one task is a waste of resources.
+
+!!! example
+    A workflow that runs a single [GROMACS][ref-uenv-gromacs] simulation, that uses one GPU.
+
+    * The optimal use of resources would allocate one quarter of a node, and allow other jobs to access the other three GPUs.
+
+    A workflow that runs 100 independent [GROMACS][ref-uenv-gromacs] simulations, where each simulation requires two GPUs.
+
+    * The optimal use of resources would allocate 50 nodes, with two simulations run on each node.
+
+[](){#ref-slurm-sharing}
+### Node sharing
+
+!!! under-construction
+    Node sharing, whereby jobs can request part of the resources on a node, and multiple jobs can run on a node (possibly from different users) is _not currently available on Alps clusters_.
+
+    CSCS will support this feature on some Alps [clusters][ref-alps-clusters] in the near-medium future.
+
+[](){#ref-slurm-exclusive}
+### Running more than one job step per node
+
+Running multiple job steps in parallel on the same allocated set of nodes can improve resource utilization by taking advantage of all the available CPUs, GPUs, or memory within a single job allocation.
+
+The approach is to:
+
+1. first allocate all the resources on each node to the job;
+2. then subdivide those resources at each invocation of srun.
+
+If Slurm believes that a request for resources (cores, gpus, memory) overlaps with what another step has already allocated, it will defer the execution until the resources are relinquished.
+This must be avoided.
+
+First ensure that *all* resources are allocated to the whole job with the following preamble:
+
+```bash title="Slurm preamble on a GH200 node"
+#!/usr/bin/env bash
+#SBATCH --exclusive --mem=450G
+```
+
+* `--exclusive` allocates all the CPUs and GPUs exclusively to this job;
+* `--mem=450G` most of allowable memory (there are 4 Grace CPUs with ~120 GB of memory on the node)
+
+!!! note
+    `--mem=0` can generally be used to allocate all memory on the node but the Slurm configuration on clariden doesn't allow this.
+
+Next, launch your applications using `srun`, carefully subdividing resources for each job step.
+The `--exclusive` flag must be used again, but note that its meaning differs in the context of `srun`.
+Here, `--exclusive` ensures that only the resources explicitly requested for a given job step are reserved and allocated to it.
+Without this flag, Slurm reserves all resources for the job step, even if it only allocates a subset -- effectively blocking further parallel `srun` invocations from accessing unrequested but needed resources.
+
+Be sure to background each `srun` command with `&`, so that subsequent job steps start immediately without waiting for previous ones to finish.
+A final `wait` command ensures that your submission script does not exit until all job steps complete.
+
+Slurm will automatically set `CUDA_VISIBLE_DEVICES` for each `srun` call, restricting GPU access to only the devices assigned to that job step.
+
+=== "single node"
+
+    !!! example "Three jobs on one node"
+        ```bash
+        #!/usr/bin/env bash
+        #SBATCH --exclusive --mem=450G
+        #SBATCH -N1
+
+        CMD="echo \$(date) \$(hostname) JobStep:\${SLURM_STEP_ID} ProcID:\${SLURM_PROCID} CUDA_VISIBLE_DEVICES=\${CUDA_VISIBLE_DEVICES}; sleep 5"
+        srun -N1 --ntasks-per-node=1 --exclusive --gpus-per-task=2 --cpus-per-gpu=5 --mem=50G --output "out-%J.log"  bash -c "${CMD}" &
+        srun -N1 --ntasks-per-node=1 --exclusive --gpus-per-task=1 --cpus-per-gpu=5 --mem=50G --output "out-%J.log"  bash -c "${CMD}" &
+        srun -N1 --ntasks-per-node=1 --exclusive --gpus-per-task=1 --cpus-per-gpu=5 --mem=50G --output "out-%J.log"  bash -c "${CMD}" &
+
+        wait
+        ```
+
+        Output (exact output will vary):
+        ```
+        $ cat out-537506.*.log
+        Tue Jul 1 11:40:46 CEST 2025 nid007104 JobStep:0 ProcID:0 CUDA_VISIBLE_DEVICES=0
+        Tue Jul 1 11:40:46 CEST 2025 nid007104 JobStep:1 ProcID:0 CUDA_VISIBLE_DEVICES=1
+        Tue Jul 1 11:40:46 CEST 2025 nid007104 JobStep:2 ProcID:0 CUDA_VISIBLE_DEVICES=2,3
+        ```
+
+
+
+=== "multi-node"
+
+    !!! example "Three jobs on two nodes"
+        ```bash
+        #!/usr/bin/env bash
+        #SBATCH --exclusive --mem=450G
+        #SBATCH -N2
+
+        CMD="echo \$(date) \$(hostname) JobStep:\${SLURM_STEP_ID} ProcID:\${SLURM_PROCID} CUDA_VISIBLE_DEVICES=\${CUDA_VISIBLE_DEVICES}; sleep 5"
+        srun -N2 --ntasks-per-node=2 --exclusive --gpus-per-task=1 --cpus-per-gpu=5 --mem=50G --output "out-%J.log"  bash -c "${CMD}" &
+        srun -N2 --ntasks-per-node=1 --exclusive --gpus-per-task=1 --cpus-per-gpu=5 --mem=50G --output "out-%J.log"  bash -c "${CMD}" &
+        srun -N2 --ntasks-per-node=1 --exclusive --gpus-per-task=1 --cpus-per-gpu=5 --mem=50G --output "out-%J.log"  bash -c "${CMD}" &
+
+        wait
+        ```
+
+        Output (exact output will vary):
+        ```
+        $ cat out-537539.*.log
+        Tue Jul 1 12:02:01 CEST 2025 nid005085 JobStep:0 ProcID:2 CUDA_VISIBLE_DEVICES=0
+        Tue Jul 1 12:02:01 CEST 2025 nid005085 JobStep:0 ProcID:3 CUDA_VISIBLE_DEVICES=1
+        Tue Jul 1 12:02:01 CEST 2025 nid005080 JobStep:0 ProcID:0 CUDA_VISIBLE_DEVICES=0
+        Tue Jul 1 12:02:01 CEST 2025 nid005080 JobStep:0 ProcID:1 CUDA_VISIBLE_DEVICES=1
+        Tue Jul 1 12:02:01 CEST 2025 nid005085 JobStep:1 ProcID:1 CUDA_VISIBLE_DEVICES=2
+        Tue Jul 1 12:02:01 CEST 2025 nid005080 JobStep:1 ProcID:0 CUDA_VISIBLE_DEVICES=2
+        Tue Jul 1 12:02:01 CEST 2025 nid005085 JobStep:2 ProcID:1 CUDA_VISIBLE_DEVICES=3
+        Tue Jul 1 12:02:01 CEST 2025 nid005080 JobStep:2 ProcID:0 CUDA_VISIBLE_DEVICES=3
+        ```

docs/software/communication/openmpi.md

@@ -22,7 +22,7 @@ OpenMPI is provided through a [uenv][ref-uenv] similar to [`prgenv-gnu`][ref-uen
 Once the uenv is loaded, compiling and linking with OpenMPI and libfabric is transparent.
 At runtime, some additional options must be set to correctly use the Slingshot network.
 
-First, when launching applications through slurm, [PMIx](https://pmix.github.com) must be used for application launching.
+First, when launching applications through Slurm, [PMIx](https://pmix.github.com) must be used for application launching.
 This is done with the `--mpi` flag of `srun`:
 ```bash
 srun --mpi=pmix ...

docs/software/sciapps/cp2k.md

@@ -65,7 +65,7 @@ On our systems, CP2K is built with the following dependencies:
 
 ### Running on the HPC platform
 
-To start a job, two bash scripts are potentially required: a [slurm] submission script, and a wrapper to start the [CUDA
+To start a job, two bash scripts are potentially required: a [Slurm] submission script, and a wrapper to start the [CUDA
 MPS] daemon so that multiple MPI ranks can use the same GPU.
 
 ```bash title="run_cp2k.sh"
@@ -138,7 +138,7 @@ sbatch run_cp2k.sh
 
     Each GH200 node has 4 modules, each of them composed of a ARM Grace CPU with 72 cores and a H200 GPU directly
     attached to it. Please see [Alps hardware][ref-alps-hardware] for more information.
-    It is important that the number of MPI ranks passed to [slurm] with `--ntasks-per-node` is a multiple of 4.
+    It is important that the number of MPI ranks passed to [Slurm] with `--ntasks-per-node` is a multiple of 4.
 
     ??? note
 
@@ -524,5 +524,5 @@ As a workaround, you can disable CUDA acceleration for the grid backend:
 [OpenBLAS]: http://www.openmathlib.org/OpenBLAS/
 [Intel MKL]: https://www.intel.com/content/www/us/en/developer/tools/oneapi/onemkl.html
 [Cray MPICH]: https://docs.nersc.gov/development/programming-models/mpi/cray-mpich/
-[slurm]: https://slurm.schedmd.com/
+[Slurm]: https://slurm.schedmd.com/
 [CUDA MPS]: https://docs.nvidia.com/deploy/mps/index.html