Deployment Infrastructure V4: Multi-HPC Deployments

[CodeGat]

References issue ACCESS-NRI/build-cd#121 and detailed PR ACCESS-NRI/build-cd#218

Important

This PR is a major update to the deployment infrastructure. See below for the prerequisites for this repository to be able to merge this PR.

Background

This update to the deployment infrastructure allows deployment to multiple different HPC systems (or the same one but a different spack instance) in parallel.

Some of the major changes relevant to a model deployment repository include:

• spack.yaml: A new way to format spack.yaml so that even if there are differences in HPC architecture or variants, we can still deploy using a single spack.yaml. See https://github.com/ACCESS-NRI/ACCESS-OM2/blob/ce5fdd27badabcbff464d50fcdc4df8e80a74123/spack.yaml for an example. You can still deploy using the traditional spack.specs[0] method.

• Choose Where To Deploy Pre/Releases: Model Deployment Repositories can now set the repo-level vars.PRERELEASE_DEPLOYMENT_TARGETS/vars.RELEASE_DEPLOYMENT_TARGETS to choose which HPCs they deploy to. This is of the form of a space-separated list of GitHub Environments (minus any potential Prerelease bit), for example: vars.PRERELEASE_DEPLOYMENT_TARGET == Gadi Setonix, vars.RELEASE_DEPLOYMENT_TARGET == Gadi.

• Updated Deployment Comments/Release Notes: Since we can deploy to multiple targets, we've had to rework the style of the deployment comments/notes. Example of Release Notes here: Make release Job Take Inputs From Multi-Target Deployments build-cd#200 (comment) and Comment here: Fix Up Multi-Target TODOs build-cd#199 (comment)

Prerequisites for Merging

• Add values for repo-level vars.[PRE]RELEASE_DEPLOYMENT_TARGETS
• Update repo-level vars.SPACK_YAML_SCHEMA_VERSION to 1-0-4
• Update entrypoints to @v4, update ci.yml pr-closed jobs model input to root-sbd, update cd.yml cd job permissions.pull-requests:write (This PR!)
• Add GitHub Environment for Releases with name SUPERCOMPUTER Release rather than SUPERCOMPUTER

[anton-seaice]

Does this need a change to the spack.yaml or does the current format work?

Can I deploy twice to Gadi with different compilers ?

[CodeGat]

The current format works fine!
And yes, technically you could - although each compiler that you would want to test with would have to be it's own GitHub Environment, which means it's own spack instance. i suppose you could set the environment to the same instance, but the issue would then be that there is nothing the demarcate the packages...I think for now it's a 'technically possible but haven't thought too deeply about that use case'

[anton-seaice]

Ok - so we should merge this ?

[CodeGat]

That would be good. It'd require a rebase to open PRs to propagate the changes from main to the PR branch, if you wanted to use it for existing PRs. Keep me in the loop regarding how it's going!

[anton-seaice]

Update with new versions from build-cd