modules: warn for modules:default:roots:tcl set and arch_folder:false

[albestro]

This is a minor bug fix, since it does not apply to any recipe currently published in https://github.com/eth-cscs/alps-uenv, but it represents a good sanity check that might help users.

Since the default configuration for spack would be to consider arch_folder: true, if the recipes don't explicitly set it, the modules get generated with an additional folder level, namely the arch_folder (e.g. /user-environment/modules/linux-sles15-neoverse_v2 instead of simply /user-environment/modules).

The problem arises with usage, because uenv sets MODULEPATH to /user-environment/modules, so the modules are not anymore reachable simply by name, but they need to be prefixed with the specific arch_folder (e.g. linux-sles15-neoverse_v2/cmake instead of cmake). That does not translates just to a burden for the user that has to prefix all module names with the arch_folder, but it actually does not work because modules dependencies in modules themselves are not prefixed with the arch_folder.

Spack implements this functionality by adding to MODULEPATH all arch_folders available on the system.

I think we can live without functionality, but we should anyway enforce it to catch earlier (at build time) potential problems (which would otherwise show up just at usage time).

EDIT: they are not strictly related, but since they act on the same portion of code, I decided to introduce dependency on #275

• only use modules.yaml from recipe #275
• Evaluate how to add extra constraint schema for modules.yaml

[msimberg]

Looks good to me.

I don't know if it might be worth expanding the messages to say something about arch_folder:true being the default? If a user leaves out arch_folder they might be confused why there's a complaint about arch_folder being enabled?

This is quite a corner case though and I think this is already very good.

[msimberg]

In fact, is this worth mentioning in the docs as well?

[albestro]

@msimberg you raise a very good point, on which I fully agree.

The point is that we just use the spack schema (and related doc), since we merely proxy it to spack with the only change of the root folder (actually, we might think of adding at least a warning for that too, because we overwrite modules:default:roots:tcl without telling).

So, what we'd like to do is to constrain the functionalities, mainly:

• not supporting arch_folder
• not supporting a custom root path for modules

One option might be to add this "extra" schema constraints as python checks on modules.yaml load, and warn the user with a message in case those keys are found to be different. Other options, where we add constraints to an existing schema (from spack), sounds like a major complication.

[albestro]

@bcumming @msimberg what do you think of the current solution?

I added a very basic schema for modules.yaml where we just care about stackinator constraints. Basically it is semantically on top of the spack one (that will be used by spack itself later), and it enforces:

• no modules:default:roots:tcl should be set, so that user is not surprised that he cannot control where modules end up
• modules:default:arch_folder is by default false, differently from spack, so that the user can just skip it. otherwise, if the user decides to specify it to true, it results in an error

If this solution is accepted, I will work on refining a couple of things, like the doc and the type of exception raised. I can also argument about a couple of design decisions.

[albestro]

FYI, I've just checked in alps-uenv and there are some recipes that believes they are setting the module path, but stackinator is currently overwriting it.

RECIPE ./mch/v8: ERROR - module path (/user-environment/modules) is not in store path (/mch-environment/v8)
RECIPE ./mch/v7: ERROR - module path (/user-environment/modules) is not in store path (/mch-environment/v7)
RECIPE ./mch/v6: ERROR - module path (/user-environment/modules) is not in store path (/mch-environment/v6)
RECIPE ./editors/25.3: ERROR - module path (/snap/modules) is not in store path (/user-tools)
RECIPE ./editors/24.7: ERROR - module path (/snap/modules) is not in store path (/user-tools)
RECIPE ./jupyterlab/v4.1.8/mc: ERROR - module path (/user-environment/modules) is not in store path (/user-tools)
RECIPE ./wcp/icon/v1/gh200: ERROR - module path (/snap/modules) is not in store path (/user-environment)
RECIPE ./wcp/icon/v1/a100: ERROR - module path (/snap/modules) is not in store path (/user-environment)

[msimberg]

Still looks good to me. One clarifying question.

[albestro]

I updated also the doc to help you understanding what's going on.

If we agree on these changes, I'm going to open a PR in alps-uenv to fix the recipes.
Specifically:

• I'm not going to remove arch_folder:false, because that would add a requirement on stackinator version used (before this PR leaving it out was an error, with this PR it becomes superfluous).
• I'm going to remove from all recipes modules:default:roots:tcl entries, since before this PR it was unnecessary, with this PR it becomes an error. (branch)

Note: I'm not enforcing anything on lmod, I don't see them used, but it might be a good idea restricting them too.

[albestro]

As per discussion about modules:default:roots:tcl with @bcumming (partially offline), we opted for a "soft" policy which still allows the specification of the field, which gets ignored by overwriting it (as currently happens), giving just a warning message about it. Related changes in the last commit cf581e7 (code, test and doc).

I'd still go for cleaning up recipes with the aforementioned branch, just to try steering future recipes (in case of copy-paste) in the new direction.

EDIT: Just opened eth-cscs/alps-uenv#276

[bcumming]

this is a more robust solution 👍