From ad2ff8146630fe07bdeff98ec52ed02b4a409947 Mon Sep 17 00:00:00 2001 From: Claudia Watson Date: Wed, 15 Oct 2025 15:17:48 +0100 Subject: [PATCH 1/7] draft of eessi docs --- docs/eessi.md | 63 +++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 63 insertions(+) create mode 100644 docs/eessi.md diff --git a/docs/eessi.md b/docs/eessi.md new file mode 100644 index 000000000..24ab12aad --- /dev/null +++ b/docs/eessi.md @@ -0,0 +1,63 @@ +# EESSI Guide + +## How to load EESSI +Loading an EESSI environment module: +```[bash] +source /cvmfs/software.eessi.io/versions/2023.06/init/lmod/bash +``` +Activating EESSI environment: +```[bash] +source /cvmfs/software.eessi.io/versions/2023.06/init/bash +``` + +## GPU Support with EESI +To enable GPU support you need a site-specific build that has `cuda` enabled. For guide to do this please refer to `docs/image-build`. This is +because CUDA-drivers are host specific and EESSI can not ship NVIDIA drivers due to licensing + kernel specific constraints. This means that the +host must provide the drivers in a known location (`host_injections`). + +### Using GPUs +All CUDA-enabled software in EESSI expects CUDA drivers in a specific `host_injections` subdirectory.
+```[bash] +ls -l /cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/amd/zen3/software/CUDA/12.1.1/bin/nvcc +``` +The output of this should show a symlink to the EESSI `host_injections` dir like so: +```[bash] +lrwxrwxrwx 1 cvmfs cvmfs 109 May 6 2024 /cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/amd/zen3/software/CUDA/12.1.1/bin/nvcc +-> /cvmfs/software.eessi.io/host_injections/2023.06/software/linux/x86_64/amd/zen3/software/CUDA/12.1.1/bin/nvcc +``` +To expose the Nvidia GPU drivers. +```[bash] +/cvmfs/software.eessi.io/versions/2023.06/scripts/gpu_support/nvidia/link_nvidia_host_libraries.sh +``` + +### Buidling with GPUs + +Checking `nvcc --version` and `which nvcc` to see if `CUDA` compiler is found.
+
+If `nvcc` not found run:
+```[bash] +export PATH=/usr/local/cuda-13.0/bin:$PATH +``` +(with your specific cuda version)
+`which nvcc` should now show path to compiler.
+
+Running `which gcc` will give path `.../2023.06/compat...`
+Loading EESSI module (It is important to load a `gcc` that is compatible with the host's CUDA version.):
+```[bash] +module load GCC/12.3.0 +``` +Now running `which gcc` will give path `.../2023.06/software...`
+
+Now you can run `cmake` and `make` to compile `CUDA` using EESSI's `gcc`.
+ +#### Test setup: Compile deviceQuery from CUDA-Samples +To test that your EESSI set up can compile `CUDA`, try compiling deviceQuery from CUDA-Samples with the following steps:
+```[bash] +git clone https://github.com/NVIDIA/cuda-samples.git +cd cuda-samples/Samples/1_Utilities/deviceQuery +mkdir -p build +cd build +cmake .. +make +./deviceQuery +``` From 524fe0f5c6dc3854abf294175f133f1d1f813015 Mon Sep 17 00:00:00 2001 From: Claudia Watson Date: Thu, 16 Oct 2025 10:12:02 +0100 Subject: [PATCH 2/7] second draft of eessi doc --- docs/eessi.md | 79 ++++++++++++++++++++++++++++----------------------- 1 file changed, 44 insertions(+), 35 deletions(-) diff --git a/docs/eessi.md b/docs/eessi.md index 24ab12aad..6e3a98f3a 100644 --- a/docs/eessi.md +++ b/docs/eessi.md @@ -1,57 +1,66 @@ # EESSI Guide -## How to load EESSI -Loading an EESSI environment module: -```[bash] -source /cvmfs/software.eessi.io/versions/2023.06/init/lmod/bash -``` -Activating EESSI environment: +## How to Load EESSI +EESSI can be initialised using the following method. + +The EESSI environment is sourced (in a non-reversible way) by running: ```[bash] source /cvmfs/software.eessi.io/versions/2023.06/init/bash ``` -## GPU Support with EESI -To enable GPU support you need a site-specific build that has `cuda` enabled. For guide to do this please refer to `docs/image-build`. This is -because CUDA-drivers are host specific and EESSI can not ship NVIDIA drivers due to licensing + kernel specific constraints. This means that the -host must provide the drivers in a known location (`host_injections`). +This is non-reversible because it: +- Changes your `$PATH`, `$MODULEPATH`, `$LD_LIBRARY_PATH`, and other critical environment variables. +- Sets EESSI-specific variables such as `EESSI_ROOT`. + +This is the recommended method because it: +- Detects your CPU architecture and OS. +- Detects and configures GPU support. +- Prepares the full EESSI software stack. +- Sets up Lmod (environment module system). + +The [EESSI docs](https://www.eessi.io/docs/using_eessi/setting_up_environment/) offer another method to load EESSI, in addition to one above. The alternative method only initialises the Lmod module system and does not load a platform-specific setup. For these reasons, it is recommened to use the method detailed above. + +Successful environment setup will show `{EESSI 2023.06}` at the start of your CLI. + +To deactivate your EESSI environment you can either restart your shell using `exec bash` or exit the shell by `exit`. + +## GPU Support with EESSI +To enable GPU support, you need a site-specific build that has CUDA enabled. For a guide on how to do this, please refer to [docs/image-build.md](../image-build.md). ### Using GPUs -All CUDA-enabled software in EESSI expects CUDA drivers in a specific `host_injections` subdirectory.
-```[bash] -ls -l /cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/amd/zen3/software/CUDA/12.1.1/bin/nvcc -``` -The output of this should show a symlink to the EESSI `host_injections` dir like so: -```[bash] -lrwxrwxrwx 1 cvmfs cvmfs 109 May 6 2024 /cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/amd/zen3/software/CUDA/12.1.1/bin/nvcc --> /cvmfs/software.eessi.io/host_injections/2023.06/software/linux/x86_64/amd/zen3/software/CUDA/12.1.1/bin/nvcc -``` -To expose the Nvidia GPU drivers. +All CUDA-enabled software in EESSI expects CUDA drivers in a specific `host_injections` directory. + +#### To expose the NVIDIA GPU drivers: +Use the `link_nvidia_host_libraries.sh` script, provided by EESSI, to symlink your GPU drivers into `host_injections`. ```[bash] /cvmfs/software.eessi.io/versions/2023.06/scripts/gpu_support/nvidia/link_nvidia_host_libraries.sh ``` +Rerun this script when your NVIDIA GPU drivers are updated. It is also safe to rerun at any time as the script will detect if the driver versions have already been symlinked. + +### Building with GPUs -### Buidling with GPUs +Run `which nvcc` to confirm that the CUDA compiler is found. -Checking `nvcc --version` and `which nvcc` to see if `CUDA` compiler is found.
-
-If `nvcc` not found run:
+If `nvcc` is not found, add the CUDA path to your environment: ```[bash] -export PATH=/usr/local/cuda-13.0/bin:$PATH +export PATH=/usr/local/cuda/bin:$PATH ``` -(with your specific cuda version)
-`which nvcc` should now show path to compiler.
-
-Running `which gcc` will give path `.../2023.06/compat...`
-Loading EESSI module (It is important to load a `gcc` that is compatible with the host's CUDA version.):
+ +`which nvcc` should now show the path to the CUDA compiler. + +#### Loading EESSI module for the GCC compiler + +Running `which gcc` with EESSI initialised should initially show a path `.../2023.06/compat...` which points to the compatibility compiler. +It is important to load a `gcc` version that is compatible with the host's CUDA version: ```[bash] module load GCC/12.3.0 ``` -Now running `which gcc` will give path `.../2023.06/software...`
-
-Now you can run `cmake` and `make` to compile `CUDA` using EESSI's `gcc`.
+Running `which gcc` will now give a path `.../2023.06/software...` which is the full compiler provided by EESSI. This is what we want for CUDA builds. + +Now you can run `cmake` and `make` to compile CUDA using EESSI's `gcc`. -#### Test setup: Compile deviceQuery from CUDA-Samples -To test that your EESSI set up can compile `CUDA`, try compiling deviceQuery from CUDA-Samples with the following steps:
+#### Test: Compile deviceQuery from CUDA-Samples +To test that your EESSI setup can compile CUDA, try compiling `deviceQuery` from CUDA-Samples with the following steps: ```[bash] git clone https://github.com/NVIDIA/cuda-samples.git cd cuda-samples/Samples/1_Utilities/deviceQuery From c225fd3d3efe759ff4a9e89d7bf529ef487d625b Mon Sep 17 00:00:00 2001 From: Claudia Watson Date: Tue, 21 Oct 2025 12:47:07 +0100 Subject: [PATCH 3/7] third draft of docs --- docs/eessi.md | 31 +++++++++++++++++++++---------- 1 file changed, 21 insertions(+), 10 deletions(-) diff --git a/docs/eessi.md b/docs/eessi.md index 6e3a98f3a..27c170a4f 100644 --- a/docs/eessi.md +++ b/docs/eessi.md @@ -1,9 +1,7 @@ # EESSI Guide ## How to Load EESSI -EESSI can be initialised using the following method. - -The EESSI environment is sourced (in a non-reversible way) by running: +The EESSI environment can be initialise by running: ```[bash] source /cvmfs/software.eessi.io/versions/2023.06/init/bash ``` @@ -12,20 +10,20 @@ This is non-reversible because it: - Changes your `$PATH`, `$MODULEPATH`, `$LD_LIBRARY_PATH`, and other critical environment variables. - Sets EESSI-specific variables such as `EESSI_ROOT`. -This is the recommended method because it: -- Detects your CPU architecture and OS. +However, it is the recommended method because it: +- Detects system CPU architecture and OS. - Detects and configures GPU support. - Prepares the full EESSI software stack. - Sets up Lmod (environment module system). -The [EESSI docs](https://www.eessi.io/docs/using_eessi/setting_up_environment/) offer another method to load EESSI, in addition to one above. The alternative method only initialises the Lmod module system and does not load a platform-specific setup. For these reasons, it is recommened to use the method detailed above. +The [EESSI docs](https://www.eessi.io/docs/using_eessi/setting_up_environment/) offer another method to load EESSI. This alternative method only initialises the Lmod module system and does not load a platform-specific setup. For these reasons, it is recommended to use the method detailed above. Successful environment setup will show `{EESSI 2023.06}` at the start of your CLI. -To deactivate your EESSI environment you can either restart your shell using `exec bash` or exit the shell by `exit`. +To deactivate your EESSI environment you can either restart your shell using `exec bash` or exit the shell by running `exit`. ## GPU Support with EESSI -To enable GPU support, you need a site-specific build that has CUDA enabled. For a guide on how to do this, please refer to [docs/image-build.md](../image-build.md). +To enable GPU support, the cluster must be running a site-specific image build that has CUDA enabled. For a guide on how to do this, please refer to [docs/image-build.md](../image-build.md). ### Using GPUs All CUDA-enabled software in EESSI expects CUDA drivers in a specific `host_injections` directory. @@ -51,10 +49,23 @@ export PATH=/usr/local/cuda/bin:$PATH #### Loading EESSI module for the GCC compiler Running `which gcc` with EESSI initialised should initially show a path `.../2023.06/compat...` which points to the compatibility compiler. -It is important to load a `gcc` version that is compatible with the host's CUDA version: + +It is important to load a `gcc` version that is compatible with the host's CUDA version. This can be found by referring to the table below: + +To find your cuda version run `nvcc --version` +| **CUDA Version** | **Recommended GCC Module (EESSI 2023.06)** | +|----------------------|---------------------------------------------------------------------| +| CUDA 13.0 | GCC/12.3.0 | +| CUDA 12.3 – 12.5 | GCC/12.3.0 or GCC/12.2.0 (officially supports GCC 12.x) | +| CUDA 12.0 – 12.2 | GCC/12.2.0 (officially listed supported version) | +| CUDA 11.4 – 11.8 | no matching GCC module in EESSI 2023.06 | +| CUDA < 11.4 | not supported | + +To load correct ESSI module run, substituing x for the compatible GCC version: ```[bash] -module load GCC/12.3.0 +module load GCC/12.x.0 ``` + Running `which gcc` will now give a path `.../2023.06/software...` which is the full compiler provided by EESSI. This is what we want for CUDA builds. Now you can run `cmake` and `make` to compile CUDA using EESSI's `gcc`. From 9a2aaee5fd7db7895da9f9d5052cfedf05e8103e Mon Sep 17 00:00:00 2001 From: Claudia Watson Date: Tue, 21 Oct 2025 12:54:05 +0100 Subject: [PATCH 4/7] Fix typo in EESSI documentation --- docs/eessi.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/eessi.md b/docs/eessi.md index 27c170a4f..c62205709 100644 --- a/docs/eessi.md +++ b/docs/eessi.md @@ -61,7 +61,7 @@ To find your cuda version run `nvcc --version` | CUDA 11.4 – 11.8 | no matching GCC module in EESSI 2023.06 | | CUDA < 11.4 | not supported | -To load correct ESSI module run, substituing x for the compatible GCC version: +To load correct ESSI module run, substituting x for the compatible GCC version: ```[bash] module load GCC/12.x.0 ``` From 5ce8e6d773d6b1e3a3714f8da99c4d65f610ab1f Mon Sep 17 00:00:00 2001 From: Claudia Watson Date: Tue, 21 Oct 2025 12:47:07 +0100 Subject: [PATCH 5/7] third draft of docs --- docs/eessi.md | 45 +++++++++++++++++++++++++++++++++++---------- 1 file changed, 35 insertions(+), 10 deletions(-) diff --git a/docs/eessi.md b/docs/eessi.md index 6e3a98f3a..05ed150e6 100644 --- a/docs/eessi.md +++ b/docs/eessi.md @@ -1,40 +1,47 @@ # EESSI Guide ## How to Load EESSI -EESSI can be initialised using the following method. -The EESSI environment is sourced (in a non-reversible way) by running: +The EESSI environment can be initialise by running: + ```[bash] source /cvmfs/software.eessi.io/versions/2023.06/init/bash ``` This is non-reversible because it: + - Changes your `$PATH`, `$MODULEPATH`, `$LD_LIBRARY_PATH`, and other critical environment variables. - Sets EESSI-specific variables such as `EESSI_ROOT`. -This is the recommended method because it: -- Detects your CPU architecture and OS. +However, it is the recommended method because it: + +- Detects system CPU architecture and OS. - Detects and configures GPU support. - Prepares the full EESSI software stack. - Sets up Lmod (environment module system). -The [EESSI docs](https://www.eessi.io/docs/using_eessi/setting_up_environment/) offer another method to load EESSI, in addition to one above. The alternative method only initialises the Lmod module system and does not load a platform-specific setup. For these reasons, it is recommened to use the method detailed above. +The [EESSI docs](https://www.eessi.io/docs/using_eessi/setting_up_environment/) offer another method to load EESSI. This alternative method only initialises the Lmod module system and does not load a platform-specific setup. For these reasons, it is recommended to use the method detailed above. Successful environment setup will show `{EESSI 2023.06}` at the start of your CLI. -To deactivate your EESSI environment you can either restart your shell using `exec bash` or exit the shell by `exit`. +To deactivate your EESSI environment you can either restart your shell using `exec bash` or exit the shell by running `exit`. ## GPU Support with EESSI -To enable GPU support, you need a site-specific build that has CUDA enabled. For a guide on how to do this, please refer to [docs/image-build.md](../image-build.md). + +To enable GPU support, the cluster must be running a site-specific image build that has CUDA enabled. For a guide on how to do this, please refer to [docs/image-build.md](../image-build.md). ### Using GPUs + All CUDA-enabled software in EESSI expects CUDA drivers in a specific `host_injections` directory. -#### To expose the NVIDIA GPU drivers: +#### To expose the NVIDIA GPU drivers + Use the `link_nvidia_host_libraries.sh` script, provided by EESSI, to symlink your GPU drivers into `host_injections`. + ```[bash] /cvmfs/software.eessi.io/versions/2023.06/scripts/gpu_support/nvidia/link_nvidia_host_libraries.sh ``` + Rerun this script when your NVIDIA GPU drivers are updated. It is also safe to rerun at any time as the script will detect if the driver versions have already been symlinked. ### Building with GPUs @@ -42,6 +49,7 @@ Rerun this script when your NVIDIA GPU drivers are updated. It is also safe to r Run `which nvcc` to confirm that the CUDA compiler is found. If `nvcc` is not found, add the CUDA path to your environment: + ```[bash] export PATH=/usr/local/cuda/bin:$PATH ``` @@ -51,16 +59,33 @@ export PATH=/usr/local/cuda/bin:$PATH #### Loading EESSI module for the GCC compiler Running `which gcc` with EESSI initialised should initially show a path `.../2023.06/compat...` which points to the compatibility compiler. -It is important to load a `gcc` version that is compatible with the host's CUDA version: + +It is important to load a `gcc` version that is compatible with the host's CUDA version. This can be found by referring to the table below: + +To find your cuda version run `nvcc --version` + +| **CUDA Version** | **Recommended GCC Module (EESSI 2023.06)** | +| ---------------- | ------------------------------------------------------- | +| CUDA 13.0 | GCC/12.3.0 | +| CUDA 12.3 – 12.5 | GCC/12.3.0 or GCC/12.2.0 (officially supports GCC 12.x) | +| CUDA 12.0 – 12.2 | GCC/12.2.0 (officially listed supported version) | +| CUDA 11.4 – 11.8 | no matching GCC module in EESSI 2023.06 | +| CUDA < 11.4 | not supported | + +To load correct ESSI module run, substituing x for the compatible GCC version: + ```[bash] -module load GCC/12.3.0 +module load GCC/12.x.0 ``` + Running `which gcc` will now give a path `.../2023.06/software...` which is the full compiler provided by EESSI. This is what we want for CUDA builds. Now you can run `cmake` and `make` to compile CUDA using EESSI's `gcc`. #### Test: Compile deviceQuery from CUDA-Samples + To test that your EESSI setup can compile CUDA, try compiling `deviceQuery` from CUDA-Samples with the following steps: + ```[bash] git clone https://github.com/NVIDIA/cuda-samples.git cd cuda-samples/Samples/1_Utilities/deviceQuery From 89817f2a53441ec203b7f7116293f94f936ca6ac Mon Sep 17 00:00:00 2001 From: Claudia Watson Date: Wed, 22 Oct 2025 13:28:48 +0100 Subject: [PATCH 6/7] final draft of eessi docs --- docs/eessi.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/eessi.md b/docs/eessi.md index 05ed150e6..865e1fc94 100644 --- a/docs/eessi.md +++ b/docs/eessi.md @@ -72,7 +72,7 @@ To find your cuda version run `nvcc --version` | CUDA 11.4 – 11.8 | no matching GCC module in EESSI 2023.06 | | CUDA < 11.4 | not supported | -To load correct ESSI module run, substituing x for the compatible GCC version: +To load correct ESSI module run, substituting x for the compatible GCC version: ```[bash] module load GCC/12.x.0 From 47a0d0a9ce1d814871ba847e725cffb242237671 Mon Sep 17 00:00:00 2001 From: Claudia Watson Date: Wed, 22 Oct 2025 13:36:46 +0100 Subject: [PATCH 7/7] fixing linting --- docs/eessi.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/docs/eessi.md b/docs/eessi.md index dc2f11a5d..865e1fc94 100644 --- a/docs/eessi.md +++ b/docs/eessi.md @@ -14,6 +14,7 @@ This is non-reversible because it: - Sets EESSI-specific variables such as `EESSI_ROOT`. However, it is the recommended method because it: + - Detects system CPU architecture and OS. - Detects and configures GPU support. - Prepares the full EESSI software stack. @@ -26,6 +27,7 @@ Successful environment setup will show `{EESSI 2023.06}` at the start of your CL To deactivate your EESSI environment you can either restart your shell using `exec bash` or exit the shell by running `exit`. ## GPU Support with EESSI + To enable GPU support, the cluster must be running a site-specific image build that has CUDA enabled. For a guide on how to do this, please refer to [docs/image-build.md](../image-build.md). ### Using GPUs