Snapshots wtih Split package repo #1123
Description
Background
Spack currently provides snapshot tags based on the last tested develop commit with all passing ✅ CI. These tags are generated weekly as develop-YYYY-MM-DD and then are used to create a snapshot build cache which is a copy the latest specs in the develop mirror. This snapshot is meant to represent the latest "working" concretization of the Spack on develop. As this is capturing a build cache, the major function of these tags is to capture a working packages repository, but it also implicitly captures the corresponding core.
The Spack packages Gitlab CI will track Spack core using an environment variable (SPACK_CHECKOUT_VERSION) that is periodically updated to a newer version of the Spack core develop branch. This variable currently has no formal procedure for how it should be updated. Initially the process will likely be a manual process of selecting a commit on Spack core develop and opening a PR that rebuilds everything once a week and is verified by a Spack packages repository maintainer.
Problem
Once the packages repository has split from Spack the relationship of snapshot tags between Spack core and Spack packages becomes more ambiguous as core is not directly connected to a specific version of the packages repository. For example, if there is a bug in Spack core that version will not be used for testing the Spack packages and the update will be deferred. However, the packages repository may still want to produce a snapshot tag.
This means that a weekly snapshot of Spack core develop would not necessarily correspond to the same weekly snapshot of the Spack packages. This inconsistency makes it somewhat confusing for users to reason about what versions of Spack core have been tested against the Spack packages repository.
Even if both Spack packages and Spack core do successfully update on the same day it is unclear if it makes sense to snapshot the Spack packages repository using a Spack core that was not used for the recipe changes from the previous week. Meaning, it would make more sense to generate a snapshot of the Spack packages repository using the previous weeks Spack core that was used for CI testing the preceding week.
Proposal
There are a many ways to proceed here, but a proposal that seem to make the most sense initially is presented here.
Only snapshot Spack packages, and then associated Spack core with that snapshot even if the commit hasn't changed or has changed multiple times in the preceding week. That is to say, when Spack packages uses a commit from Spack core for generating its snapshot, push a corresponding snapshot tag to corresponding commit on Spack core.
This solves the issue of diverging snapshot tags, and seems to preserve the intent of the snapshots which is to capture the state of the stacks built in CI at a point in time. It also makes it easier to search for the version of Spack core associated with a snapshot and lets the Spack packages repository move independently as needed.