add generic hooks for unsupported modules #91

bedroge · 2025-09-25T11:37:40Z

This makes the existing zen4 pre_fetch / pre_module / post_module hooks a bit more generic, so that they can also be used for adding dummy modulefiles for other unsupported modules. The function is_unsupported_module is used to determine if something is unsupported, and if so, which environment variable can be used to ignore the LmodError of that module.

The hooks are also applied to make a module file with an LmodError for Rust 1.65.0 on A64FX.

Tested locally by building Rust 1.65.0 for A64FX with EESSI-extend:

$ eb Rust-1.65.0-GCCcore-12.2.0.eb
== Temporary log file in case of crash /tmp/eb-c40k2fnp/easybuild-5vbxdeyd.log
...
== processing EasyBuild easyconfig /home/bob/easybuild/easybuild-easyconfigs/easybuild/easyconfigs/r/Rust/Rust-1.65.0-GCCcore-12.2.0.eb
== building and installing Rust/1.65.0-GCCcore-12.2.0...
  >> installation prefix: /home/bob/eessi/versions/2023.06/software/linux/aarch64/a64fx/software/Rust/1.65.0-GCCcore-12.2.0
== fetching files and verifying checksums...
== Running pre-fetch hook...
== Updated build option 'module-only' to 'True'
== Updated build option 'force' to 'True'
  >> sources:
  >> /data/eb/sources/r/Rust/rustc-1.65.0-src.tar.gz [SHA256: 5828bb67f677eabf8c384020582b0ce7af884e1c84389484f7f8d00dd82c0038]
  >> patches:
  >> /home/bob/easybuild/easybuild-easyconfigs/easybuild/easyconfigs/r/Rust/Rust-1.60_sysroot-fix-interpreter.patch [SHA256: b59ed4c2591fc9098277299be21dd6752654f6f193d8652b7d21cb0fa0dd8716]
== ... (took < 1 sec)
== creating build dir, resetting environment...
  >> build dir: /data/eb/build/Rust/1.65.0/GCCcore-12.2.0
== Running post-ready hook...

WARNING: Deprecated functionality, will no longer work in EasyBuild v6.0: Easyconfig parameter 'parallel' is deprecated, use 'max_parallel' or the parallel property instead.; see https://docs.easybuild.io/deprecated-functionality/ for more 
information


WARNING: Deprecated functionality, will no longer work in EasyBuild v6.0: Easyconfig parameter 'parallel' is deprecated, use 'max_parallel' or the parallel property instead.; see https://docs.easybuild.io/deprecated-functionality/ for more 
information

== limiting parallelism to 2 (was 6, derived parallelism 8) for Rust on aarch64/a64fx 
== ... (took < 1 sec)
== unpacking [skipped]
== patching [skipped]
== preparing...
== Running pre-prepare hook...
  >> loading toolchain module: GCCcore/12.2.0
  >> loading modules for build dependencies:
  >>  * CMake/3.24.3-GCCcore-12.2.0
  >>  * Python/3.10.8-GCCcore-12.2.0-bare
  >>  * pkgconf/1.9.3-GCCcore-12.2.0
  >> loading modules for (runtime) dependencies:
  >>  * OpenSSL/1.1
  >> defining build environment for GCCcore/12.2.0 toolchain
== Running post-prepare hook...
== ... (took 1 secs)
== configuring [skipped]
== building [skipped]
== testing [skipped]
== installing [skipped]
== taking care of extensions [skipped]
== restore after iterating...
== ... (took < 1 sec)
== postprocessing [skipped]
== sanity checking [skipped]
== cleaning up [skipped]
== creating module...
== Running pre-module hook...
== Setting EESSI_IGNORE_LMOD_ERROR_A64FX_RUST1650 in initial environment
  >> generating module file @ /home/bob/eessi/versions/2023.06/software/linux/aarch64/a64fx/modules/all/Rust/1.65.0-GCCcore-12.2.0.lua
== Running post-module hook...
== Restored original build option 'module_only' to False
== Restored original build option 'force' to False
== Removing EESSI_IGNORE_LMOD_ERROR_A64FX_RUST1650 in initial environment
== ... (took < 1 sec)
== permissions...
== ... (took < 1 sec)
== packaging...
== ... (took < 1 sec)
  >> running shell command:
        bzip2 /home/bob/eessi/versions/2023.06/software/linux/aarch64/a64fx/software/Rust/1.65.0-GCCcore-12.2.0/easybuild/easybuild-Rust-1.65.0-20250925.133516.log
        [started at: 2025-09-25 13:35:16]
        [working dir: /home/bob/gits/EESSI/software-layer-scripts]
        [output and state saved to /tmp/eb-c40k2fnp/run-shell-cmd-output/bzip2-7wc0hadt]
  >> command completed: exit 0, ran in < 1s
== COMPLETED: Installation ended successfully (took 3 secs)
== Results of the build can be found in the log file(s) /home/bob/eessi/versions/2023.06/software/linux/aarch64/a64fx/software/Rust/1.65.0-GCCcore-12.2.0/easybuild/easybuild-Rust-1.65.0-20250925.133516.log.bz2
== Running post-easyblock hook...

== Build succeeded for 1 out of 1
== Summary:
   * [SUCCESS] Rust/1.65.0-GCCcore-12.2.0
== Temporary log file(s) /tmp/eb-c40k2fnp/easybuild-5vbxdeyd.log* have been removed.
== Temporary directory /tmp/eb-c40k2fnp has been removed.


$ tail -n2 ~/eessi/versions/2023.06/software/linux/aarch64/a64fx/modules/all/Rust/1.65.0-GCCcore-12.2.0.lua 
-- Built with EasyBuild version 5.1.1
if (not os.getenv("EESSI_IGNORE_LMOD_ERROR_A64FX_RUST1650")) then LmodError("Rust 1.65.0 is not supported on A64FX. Please use version 1.75.0 instead.") end

And also tested by building a 2022b easyconfig for Zen4:

$ eb -f ~/easybuild/easybuild-easyconfigs/easybuild/easyconfigs/f/file/file-5.43-GCCcore-12.2.0.eb 
== Temporary log file in case of crash /tmp/eb-91skhb9h/easybuild-kbx7d5vo.log
== processing EasyBuild easyconfig /home/bob/easybuild/easybuild-easyconfigs/easybuild/easyconfigs/f/file/file-5.43-GCCcore-12.2.0.eb
== building and installing file/5.43-GCCcore-12.2.0...
  >> installation prefix: /home/bob/eessi/versions/2023.06/software/linux/x86_64/amd/zen4/software/file/5.43-GCCcore-12.2.0
== fetching files and verifying checksums...
== Running pre-fetch hook...
== Updated build option 'module-only' to 'True'
== Updated build option 'force' to 'True'
  >> sources:
  >> /data/eb/sources/f/file/file-5.43.tar.gz [SHA256: 8c8015e91ae0e8d0321d94c78239892ef9dbc70c4ade0008c0e95894abfb1991]
== ... (took < 1 sec)
== creating build dir, resetting environment...
  >> build dir: /data/eb/build/file/5.43/GCCcore-12.2.0
== Running post-ready hook...

WARNING: Deprecated functionality, will no longer work in EasyBuild v6.0: Easyconfig parameter 'parallel' is deprecated, use 'max_parallel' or the parallel property instead.; see https://docs.easybuild.io/deprecated-functionality/ for more 
information

== ... (took < 1 sec)
== unpacking [skipped]
== patching [skipped]
== preparing...
== Running pre-prepare hook...
  >> loading toolchain module: GCCcore/12.2.0
  >> defining build environment for GCCcore/12.2.0 toolchain
== Running post-prepare hook...
== ... (took < 1 sec)
== configuring [skipped]
== building [skipped]
== testing [skipped]
== installing [skipped]
== taking care of extensions [skipped]
== restore after iterating...
== ... (took < 1 sec)
== postprocessing [skipped]
== sanity checking [skipped]
== cleaning up [skipped]
== creating module...
== Running pre-module hook...
== Setting EESSI_IGNORE_LMOD_ERROR_ZEN4_GCC1220 in initial environment
  >> generating module file @ /home/bob/eessi/versions/2023.06/software/linux/x86_64/amd/zen4/modules/all/file/5.43-GCCcore-12.2.0.lua
== Running post-module hook...
== Restored original build option 'module_only' to False
== Restored original build option 'force' to True
== Removing EESSI_IGNORE_LMOD_ERROR_ZEN4_GCC1220 in initial environment
== ... (took < 1 sec)
== permissions...
== ... (took < 1 sec)
== packaging...
== ... (took < 1 sec)
  >> running shell command:
        bzip2 /home/bob/eessi/versions/2023.06/software/linux/x86_64/amd/zen4/software/file/5.43-GCCcore-12.2.0/easybuild/easybuild-file-5.43-20250925.130223.log
        [started at: 2025-09-25 13:02:23]
        [working dir: /home/bob/gits/EESSI/software-layer-scripts]
        [output and state saved to /tmp/eb-91skhb9h/run-shell-cmd-output/bzip2-s98xawv1]
  >> command completed: exit 0, ran in < 1s
== COMPLETED: Installation ended successfully (took 1 secs)
== Results of the build can be found in the log file(s) /home/bob/eessi/versions/2023.06/software/linux/x86_64/amd/zen4/software/file/5.43-GCCcore-12.2.0/easybuild/easybuild-file-5.43-20250925.130223.log.bz2
== Running post-easyblock hook...

== Build succeeded for 1 out of 1
== Summary:
   * [SUCCESS] file/5.43-GCCcore-12.2.0
== Temporary log file(s) /tmp/eb-91skhb9h/easybuild-kbx7d5vo.log* have been removed.
== Temporary directory /tmp/eb-91skhb9h has been removed.


$ tail -n 2 ~/eessi/versions/2023.06/software/linux/x86_64/amd/zen4/modules/all/file/5.43-GCCcore-12.2.0.lua 
-- Built with EasyBuild version 5.1.1
if (not os.getenv("EESSI_IGNORE_LMOD_ERROR_ZEN4_GCC1220")) then LmodError("EasyConfigs using toolchains based on GCCcore-12.2.0 are not supported for the Zen4 architecture.\nSee https://www.eessi.io/docs/known_issues/eessi-2023.06/#gcc-1220-and-foss-2022b-based-modules-cannot-be-loaded-on-zen4-architecture") end

…rted modules

eb_hooks.py

…nstead of having them in the generic hook

…tware-layer-scripts into hook_for_unsupported_modules

ocaisa

LGTM, tested for the Zen4 case

ocaisa · 2025-09-26T11:20:15Z

bot: build repo:eessi.io-2023.06-software instance:eessi-bot-mc-aws for:arch=x86_64/amd/zen2
bot: build repo:eessi.io-2025.06-software instance:eessi-bot-mc-aws for:arch=x86_64/amd/zen2

eessi-bot-aws · 2025-09-26T11:20:22Z

New job on instance eessi-bot-mc-aws for repository eessi.io-2023.06-software
Building on: amd-zen2
Building for: x86_64/amd/zen2
Job dir: /project/def-users/SHARED/jobs/2025.09/pr_91/92593

date	job status	comment
Sep 26 11:20:21 UTC 2025	submitted	job id `92593` awaits release by job manager
Sep 26 11:21:58 UTC 2025	released	job awaits launch by Slurm scheduler
Sep 26 11:59:02 UTC 2025	running	job `92593` is running
Sep 26 12:02:20 UTC 2025	finished	😁 SUCCESS (click triangle for details) Details ✅ job output file `slurm-92593.out` ✅ no message matching `FATAL:` ✅ no message matching `ERROR:` ✅ no message matching `FAILED:` ✅ no message matching `required modules missing:` ✅ found message(s) matching `No missing installations` ✅ found message matching `.tar.gz created!` Artefacts `eessi-2023.06-software-linux-x86_64-amd-zen2-17588878880.tar.gz` size: 0 MiB (21344 bytes) entries: 1 modules under 2023.06/software/linux/x86_64/amd/zen2/modules/all no module files in tarball software under 2023.06/software/linux/x86_64/amd/zen2/software no software packages in tarball reprod directories under 2023.06/software/linux/x86_64/amd/zen2/reprod no reprod directories in tarball other under 2023.06/software/linux/x86_64/amd/zen2 `2023.06/init/easybuild/eb_hooks.py`
Sep 26 12:02:20 UTC 2025	test result	😁 SUCCESS (click triangle for details) ReFrame Summary [ OK ] ( 1/10) EESSI_LAMMPS_lj %device_type=cpu %module_name=LAMMPS/29Aug2024-foss-2023b-kokkos %scale=1_node /aeb2d9df @BotBuildTests:x86_64_amd_zen2+default P: perf: 441.741 timesteps/s (r:0, l:None, u:None) [ OK ] ( 2/10) EESSI_LAMMPS_lj %device_type=cpu %module_name=LAMMPS/2Aug2023_update2-foss-2023a-kokkos %scale=1_node /04ff9ece @BotBuildTests:x86_64_amd_zen2+default P: perf: 441.638 timesteps/s (r:0, l:None, u:None) [ OK ] ( 3/10) EESSI_OSU_coll %benchmark_info=mpi.collective.osu_allreduce %module_name=OSU-Micro-Benchmarks/7.2-gompi-2023b %scale=1_node %device_type=cpu /775175bf @BotBuildTests:x86_64_amd_zen2+default P: latency: 1.86 us (r:0, l:None, u:None) [ OK ] ( 4/10) EESSI_OSU_coll %benchmark_info=mpi.collective.osu_allreduce %module_name=OSU-Micro-Benchmarks/7.1-1-gompi-2023a %scale=1_node %device_type=cpu /52707c40 @BotBuildTests:x86_64_amd_zen2+default P: latency: 1.88 us (r:0, l:None, u:None) [ OK ] ( 5/10) EESSI_OSU_coll %benchmark_info=mpi.collective.osu_alltoall %module_name=OSU-Micro-Benchmarks/7.2-gompi-2023b %scale=1_node %device_type=cpu /b1aacda9 @BotBuildTests:x86_64_amd_zen2+default P: latency: 4.06 us (r:0, l:None, u:None) [ OK ] ( 6/10) EESSI_OSU_coll %benchmark_info=mpi.collective.osu_alltoall %module_name=OSU-Micro-Benchmarks/7.1-1-gompi-2023a %scale=1_node %device_type=cpu /c6bad193 @BotBuildTests:x86_64_amd_zen2+default P: latency: 4.49 us (r:0, l:None, u:None) [ OK ] ( 7/10) EESSI_OSU_pt2pt_CPU %benchmark_info=mpi.pt2pt.osu_latency %module_name=OSU-Micro-Benchmarks/7.2-gompi-2023b %scale=1_node /15cad6c4 @BotBuildTests:x86_64_amd_zen2+default P: latency: 0.59 us (r:0, l:None, u:None) [ OK ] ( 8/10) EESSI_OSU_pt2pt_CPU %benchmark_info=mpi.pt2pt.osu_latency %module_name=OSU-Micro-Benchmarks/7.1-1-gompi-2023a %scale=1_node /6672deda @BotBuildTests:x86_64_amd_zen2+default P: latency: 0.55 us (r:0, l:None, u:None) [ OK ] ( 9/10) EESSI_OSU_pt2pt_CPU %benchmark_info=mpi.pt2pt.osu_bw %module_name=OSU-Micro-Benchmarks/7.2-gompi-2023b %scale=1_node /2a9a47b1 @BotBuildTests:x86_64_amd_zen2+default P: bandwidth: 7312.6 MB/s (r:0, l:None, u:None) [ OK ] (10/10) EESSI_OSU_pt2pt_CPU %benchmark_info=mpi.pt2pt.osu_bw %module_name=OSU-Micro-Benchmarks/7.1-1-gompi-2023a %scale=1_node /1b24ab8e @BotBuildTests:x86_64_amd_zen2+default P: bandwidth: 7194.91 MB/s (r:0, l:None, u:None) [ PASSED ] Ran 10/10 test case(s) from 10 check(s) (0 failure(s), 0 skipped, 0 aborted) Details ✅ job output file `slurm-92593.out` ✅ no message matching `ERROR:` ✅ no message matching `[\sFAILED\s].Ran . test case`
Sep 26 12:10:17 UTC 2025	uploaded	transfer of `eessi-2023.06-software-linux-x86_64-amd-zen2-17588878880.tar.gz` to S3 bucket succeeded

eessi-bot-aws · 2025-09-26T11:20:26Z

New job on instance eessi-bot-mc-aws for repository eessi.io-2025.06-software
Building on: amd-zen2
Building for: x86_64/amd/zen2
Job dir: /project/def-users/SHARED/jobs/2025.09/pr_91/92594

date	job status	comment
Sep 26 11:20:25 UTC 2025	submitted	job id `92594` awaits release by job manager
Sep 26 11:21:56 UTC 2025	released	job awaits launch by Slurm scheduler
Sep 26 11:57:30 UTC 2025	running	job `92594` is running
Sep 26 11:59:34 UTC 2025	finished	😁 SUCCESS (click triangle for details) Details ✅ job output file `slurm-92594.out` ✅ no message matching `FATAL:` ✅ no message matching `ERROR:` ✅ no message matching `FAILED:` ✅ no message matching `required modules missing:` ✅ found message(s) matching `No missing installations` ✅ found message matching `.tar.gz created!` Artefacts `eessi-2025.06-software-linux-x86_64-amd-zen2-17588878410.tar.gz` size: 0 MiB (21346 bytes) entries: 1 modules under 2025.06/software/linux/x86_64/amd/zen2/modules/all no module files in tarball software under 2025.06/software/linux/x86_64/amd/zen2/software no software packages in tarball reprod directories under 2025.06/software/linux/x86_64/amd/zen2/reprod no reprod directories in tarball other under 2025.06/software/linux/x86_64/amd/zen2 `2025.06/init/easybuild/eb_hooks.py`
Sep 26 11:59:34 UTC 2025	test result	😢 FAILURE (click triangle for details) Reason EESSI test suite was not run, test step itself failed to execute. Details ✅ job output file `slurm-92594.out` ❌ found message matching `ERROR:` ✅ no message matching `[\sFAILED\s].Ran . test case`
Sep 26 12:10:25 UTC 2025	uploaded	transfer of `eessi-2025.06-software-linux-x86_64-amd-zen2-17588878410.tar.gz` to S3 bucket succeeded

bedroge and others added 2 commits September 25, 2025 13:31

add LmodError for Rust 1.65.0 on A64FX, make generic hook for unsuppo…

f6bb069

…rted modules

Merge branch 'main' into hook_for_unsupported_modules

2dd23ff

ocaisa reviewed Sep 26, 2025

View reviewed changes

eb_hooks.py Outdated Show resolved Hide resolved

bedroge added 2 commits September 26, 2025 12:26

move check for unsupported modules to the hook function themselves, i…

7b8a935

…nstead of having them in the generic hook

Merge branch 'hook_for_unsupported_modules' of github.com:bedroge/sof…

4bfd2cc

…tware-layer-scripts into hook_for_unsupported_modules

ocaisa approved these changes Sep 26, 2025

View reviewed changes

ocaisa added the bot:deploy label Sep 26, 2025

bedroge mentioned this pull request Sep 26, 2025

{2023.06}[a64fx,2022b] Rust 1.65.0 EESSI/software-layer#1204

Merged

ocaisa merged commit 5cccf70 into EESSI:main Sep 26, 2025
66 of 68 checks passed

bedroge deleted the hook_for_unsupported_modules branch September 26, 2025 13:12

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

add generic hooks for unsupported modules #91

add generic hooks for unsupported modules #91

Uh oh!

bedroge commented Sep 25, 2025

Uh oh!

Uh oh!

ocaisa left a comment

Uh oh!

ocaisa commented Sep 26, 2025

Uh oh!

eessi-bot-aws bot commented Sep 26, 2025 •

edited

Loading

Uh oh!

eessi-bot-aws bot commented Sep 26, 2025 •

edited

Loading

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

add generic hooks for unsupported modules #91

add generic hooks for unsupported modules #91

Uh oh!

Conversation

bedroge commented Sep 25, 2025

Uh oh!

Uh oh!

ocaisa left a comment

Choose a reason for hiding this comment

Uh oh!

ocaisa commented Sep 26, 2025

Uh oh!

eessi-bot-aws bot commented Sep 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

eessi-bot-aws bot commented Sep 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

eessi-bot-aws bot commented Sep 26, 2025 •

edited

Loading

eessi-bot-aws bot commented Sep 26, 2025 •

edited

Loading