-
Notifications
You must be signed in to change notification settings - Fork 56
New and chained environments for existing sites
The following instructions cover installing new spack environments on a pre-configured site, adding new packages to existing environments, and extending existing environments through environment chaining.
The following instructions apply to the basic environments (unified-dev, skylab-dev, neptune-dev).
git clone --recurse-submodules https://github.com/jcsda/spack-stack
cd spack-stack
Ensure Python 3.6+ is available and the default before sourcing spack. Note this is only used for building the environment. Once the environment is built, spack-stack provides the proper python executable which needs to be utilized to build applications with the newly created environment.
Source spack from submodule, set ${SPACK_STACK_DIR}, and more.
source setup.sh
See a list of sites and templates.
spack stack create env -h
Create a pre-configured Spack environment (copies site-specific, application-specific, and common config files into the environment directory). Depending on the site, the --compiler argument may require a specific version, for example --compiler=oneapi-2025.3.0 (see packages_*.yaml in the site directory).
spack stack create env --site=hera --template=unified-dev --name=unified-dev.hera.intel --compiler=intel
Activate the newly created environment Optional: decorate the command line prompt using -p. In some cases, this can mess up long lines in bash, because color codes are not escaped correctly. In this case, use export SPACK_COLOR='never' first.
cd envs/unified-dev.hera.intel/
spack env activate [-p] .
Optionally edit config files (spack.yaml, packages.yaml, modules.yaml, ...) with your editor of choice.
Process/concretize the specs; optionally check for duplicate packages.
spack concretize --force --fresh
${SPACK_STACK_DIR}/util/show_duplicate_packages.py
Install the environment; use --source to install the source code with the compiled binary package. If a build cache is available for the system (check with spack mirror list), add --no-check-signature to install from the build cache when possible, or --no-cache to install from source.
spack install [--source] [--verbose] [--fail-fast] [--no-check-signature|--no-cache]
Create lua or tcl module files (check site config modules.yaml which one is being used).
spack module [lmod|tcl] refresh
Create meta-modules for compiler and mpi (stack-<compiler>/<version>, stack-<mpi>/<version>).
spack stack setup-meta-modules
Tip. You may want to capture the output from spack concretize and spack install in log files. For example:
spack concretize ... 2>&1 | tee log.concretize
spack install ... 2>&1 | tee log.install
New packages can be added to newly created environments in the same way as for chained environments, either via the command line or by editing the environment's spack.yaml. See the next section for details.
Releases of spack-stack are deployed quarterly on supported platforms. Between releases, additional or modified packages may be installed by the spack-stack maintainers or advanced users. While some environment modifications, for example a new compiler, will require installing entire environments as described in the previous section, in many cases chained environments are sufficient. These chained environments leverage Spack's environment chaining capabilities. This mechanism allows parts of the stack to be replaced, and a new chained environment created while leaving the base release environment untouched.
A chained environment is a Spack environment ("downstream") that builds on top of one or more existing environments ("upstream"). Instead of creating a standalone installation, the downstream environment reuses the install tree(s) of previously deployed environments. The upstream environment provides a fully configured collection of packages and modules, which the downstream environment can incorporate without rebuilding them.
In a chained environment, users may override specific packages (for example, to test a newer version) or add new packages on top of those supplied by the upstream stack. Because the downstream environment references the upstream installation without modifying it, the result is a chain of environments: the new environment extends or customizes the existing software stack while preserving the stability and integrity of the base installation.
To install an additional environment within an official spack-stack installation space, change into the appropriate spack-stack root directory and run: source setup.sh before proceeding with the steps below. To create a chained environment in a personal space outside an official installation, it is recommended that you use the same spack-stack release as the one providing the upstream environment.
For example, if you are targeting an environment installed under spack-stack-1.4.1/envs, clone and check out the matching release branch:
git clone --recurse-submodules -b release/1.4.1 https://github.com/jcsda/spack-stack spack-stack-1.4.1
cd spack-stack-1.4.1
source setup.sh
Then proceed with environment creation.
The example below demonstrates creating a netcdf-test chained environment on Hera using an existing GCC-based upstream environment. In this example, a newer version of NetCDF-C is built with the GCC compiler along with all dependencies required by the UFS Weather Model.
It is recommended to use one or more meta-modules (e.g., ufs-weather-model-env) rather than individual packages when starting with the empty environment template. Meta-modules ensure that all required modules from the upstream environment are made available in the downstream environment when
spack module lmod refresh --upstream-modules
is invoked. An alternative is to start with the same template that the upstream environment is using. Finally, make sure to use the same compiler family as the upstream environment unless you intentionally override it and rebuild all dependent packages.
spack stack create env --name netcdf-test --template empty --site hera --compiler gcc \
--upstream /scratch1/NCEPDEV/nems/role.epic/spack-stack/spack-stack-1.4.1/envs/ue-gcc/install \
[--upstream /path/to/second/install] [--modify-pkg netcdf-c]
cd envs/netcdf-test
spack env activate .
When a chained environment is created, it includes an upstreams section at the end of its spack.yaml. This section records the name of each upstream environment and the location of its install tree. For example:
upstreams:
my-base-env:
install_tree: /full/path/to/spack-stack/envs/my-base-env/install
This configuration instructs Spack to reuse packages already installed in the upstream environment unless overridden in the downstream environment. Multiple upstream environments may be listed; Spack will search them in order during concretization.
Once the chained environment has been created and activated, additional packages may be added either directly from the command line, or by editing spack.yaml.
Adding Packages from the Command Line
The following command adds the ufs-weather-model-env meta-environment and requests a new version of NetCDF-C:
spack add ufs-weather-model-env%intel ^netcdf-c@4.9.2
Adding packages by editing spack.yaml
Users may alternatively edit spack.yaml (and configuration files under ./site/ and ./common/, if needed) in the same way as for a base environment installations. This approach is useful when adding multiple packages or making broader specification adjustments.
Example of a specs entry:
specs:
- ufs-weather-model-env%gcc ^netcdf-c@4.9.2
- netcdf-fortran@4.6.1 %gcc
- nco@5.0.6 %gcc
After adding new or modifying packages in the chained environment definition, proceed with concretization and installation. Use the `--upstream-modules`` flag when refreshing the module files:
spack concretize --force --fresh 2>&1 | tee log.concretize
spack install [--verbose] [--no-check-signature|--no-cache] 2>&1 | tee log.install
spack module [lmod|tcl] refresh --upstream-modules
spack stack setup-meta-modules
To use the chained environment, access it in the same way as a regular spack-stack installation: add the directory returned by spack stack setup-meta-modules (ending in */modulefiles/Core) to your $MODULEPATH.
This ensures that the correct combination of upstream and downstream module files is available, while avoiding conflicts.
Do not add the upstream environment’s module directory directly to $MODULEPATH.
Note. The --upstream option for the spack stack create env command adds a specified spack-stack installation path as an upstream environment in the resulting spack.yaml and may be invoked multiple times. The command will warn but not fail if an invalid directory is provided. If the path does not exist, check for typos and ensure you are using the correct path to the upstream environment, in particular that the install suffix is present. The command will also warn but not fail if the environment configuration files in common/ or site/ differ between the upstream environment and the chained environment. In this case, carefully check for differences between these files.
Note. The --modify-pkg option for the spack stack create env command should be used by spack-stack maintainers when a package recipe needs to be modified between-release deployments (i.e., chained environments within an official release). This option creates a separate custom Spack repository under $SPACK_ENV/envrepo/, updates the environment’s repo configuration automatically, and prevents accidental modifications to the upstream installation.
Additional guidance and cautions on chaining Spack environments can be found in the Spack documentation. In particular, avoid deleting module files or dependencies from the upstream environment.