Bootstrap: stale CI cache makes bootstrap.py mix Cabal-syntax-3.18.1.0 and Cabal-syntax-3.19.0.0 in the bootstrap package DB

[zlonast]

Describe the bug
The Bootstrap workflow fails on the 3.18 release branch (e.g. mergify backport branches) with:

Warning:
    This package indirectly depends on multiple versions of the same package. This is very likely to cause a compile failure.
      package hooks-exe (hooks-exe-3.18-22KHzF4sUT38fGhUvzyePu) requires Cabal-syntax-3.18.1.0-8zQAqSUiwaZ17MHcGFOcie
      package cabal-install-solver (cabal-install-solver-3.18.1.0-5LuTmwczm6yHss5JDfGG3K) requires Cabal-syntax-3.18.1.0-8zQAqSUiwaZ17MHcGFOcie
      package cabal-install (cabal-install-3.18.1.0) requires Cabal-syntax-3.18.1.0-8zQAqSUiwaZ17MHcGFOcie
      package Cabal (Cabal-3.18.1.0-5o8LiszPwViDtqOTbtNLRM) requires Cabal-syntax-3.18.1.0-8zQAqSUiwaZ17MHcGFOcie
      package hackage-security (hackage-security-0.6.3.3-HnzyDDEGWK8KBPRiLAacC2) requires Cabal-syntax-3.19.0.0-GjmW8g1FynRA1psf3P3WKB

followed by a compile failure like Couldn't match type 'Distribution.Types.PackageId.PackageIdentifier' ....

The root cause: bootstrap.py only initialises the bootstrap package database if it does not exist yet:

if not PKG_DB.exists():
    ...
    subprocess_run([ghc.ghc_pkg_path, 'init', PKG_DB])

When actions/cache restores a _build directory produced on a different branch (where the local packages have different versions), the stale registrations survive: local Cabal-syntax-3.18.1.0 gets registered alongside the cached Cabal-syntax-3.19.0.0. Setup.hs then resolves the newest version for every package whose bounds are loose enough — currently only hackage-security — so it is compiled against Cabal-syntax-3.19.0.0 while everything else is compiled against Cabal-syntax-3.18.1.0.

The caches collide because the cache key in .github/workflows/bootstrap.yml is

key: bootstrap-${{ ... }}-${{ matrix.ghc }}-${{ hashFiles(format('bootstrap/linux-{0}.json', matrix.ghc)) }}-${{ github.sha }}
restore-keys: bootstrap-${{ ... }}-${{ matrix.ghc }}-${{ hashFiles(...) }}-

and the bootstrap/linux-*.json files are byte-identical between master (local packages = 3.19.0.0) and 3.18 (local packages = 3.18.1.0), since local packages are only referenced by name in the plans.

To Reproduce

$ python3 bootstrap/bootstrap.py -d bootstrap/linux-9.6.7.json fetch
$ python3 bootstrap/bootstrap.py --bootstrap-sources bootstrap-sources.tar.gz

1. Run the Bootstrap workflow on master (local Cabal/Cabal-syntax/cabal-install-solver are 3.19.0.0) → _build incl. packages.conf is cached.
2. Run the Bootstrap workflow on the 3.18 branch → restore-keys matches the master cache (identical JSON hash) → stale _build/packages.conf containing Cabal-syntax-3.19.0.0 is restored.
3. bootstrap.py skips ghc-pkg init and registers local Cabal-syntax-3.18.1.0 next to it.
4. hackage-security-0.6.3.3 (the only package with bounds loose enough to admit both) is configured against 3.19.0.0, everything else against 3.18.1.0 → the warning above, then a PackageIdentifier type mismatch.
5. Example failing run: Bootstrap / ubuntu-latest ghc-9.6.7 (job 99738568748)

Expected behavior
Bootstrapping should produce a consistent package database regardless of the previous content of _build — a stale or foreign package DB must never result in multiple versions of the same package being mixed.

System information

• Operating system: GitHub Actions runner, ubuntu-latest
• cabal: built from source — 3.18.1.0 (release branch 3.18 / backport branch mergify/bp/3.18/pr-12276), 3.19.0.0 on master
• ghc: 9.6.7, 9.8.4, 9.10.3, 9.12.4 (all matrix jobs are affected)

Additional context
Proposed fix — recreate the bootstrap package database from scratch on every run, making bootstrap.py idempotent w.r.t. a dirty/polluted cache:

diff --git a/bootstrap/bootstrap.py b/bootstrap/bootstrap.py
index 5638b1364..19e88c291 100755
--- a/bootstrap/bootstrap.py
+++ b/bootstrap/bootstrap.py
@@ -275,10 +275,16 @@ UnitId = NewType('UnitId', str)
 PlanUnit = NewType('UnitId', str)
 
 def bootstrap(info: BootstrapInfo, ghc: Compiler) -> None:
-    if not PKG_DB.exists():
-        print(f'Creating package database {PKG_DB}')
-        PKG_DB.parent.mkdir(parents=True, exist_ok=True)
-        subprocess_run([ghc.ghc_pkg_path, 'init', PKG_DB])
+    # Always recreate the package database from scratch: a stale database
+    # (e.g. restored from a CI cache built from a different branch) can
+    # contain other versions of the local packages, which would lead to
+    # mixing multiple versions of the same package and compile failures.
+    if PKG_DB.exists():
+        print(f'Deleting stale package database {PKG_DB}')
+        shutil.rmtree(PKG_DB)
+    print(f'Creating package database {PKG_DB}')
+    PKG_DB.parent.mkdir(parents=True, exist_ok=True)
+    subprocess_run([ghc.ghc_pkg_path, 'init', PKG_DB])
 
     for dep in info.builtin:
         check_builtin(dep, ghc)

[zlonast]

Include the local packages in the CI cache key

bootstrap.py alone restores correctness, but the 3.18 runs would keep restoring a useless master cache. Since local packages are only referenced by name in the plans, the .cabal files of the packages that get built from the repo are now hashed into the cache key. Caches are then only shared between branches that build the same packages (master 3.19.0.0 ≠ 3.18 3.18.1.0 → no cross-branch restore), while PRs still inherit the cache of their base branch.

diff --git a/.github/workflows/bootstrap.yml b/.github/workflows/bootstrap.yml
index 78f82fe2e..9cc48feaf 100644
--- a/.github/workflows/bootstrap.yml
+++ b/.github/workflows/bootstrap.yml
@@ -44,8 +44,8 @@ jobs:
         id: bootstrap-cache
         with:
           path: "/home/runner/work/cabal/cabal/_build"
-          key: bootstrap-${{ steps.get-osver.outputs.osver }}-${{ matrix.ghc }}-${{ hashFiles(format('bootstrap/linux-{0}.json', matrix.ghc)) }}-${{ github.sha }}
-          restore-keys: bootstrap-${{ steps.get-osver.outputs.osver }}-${{ matrix.ghc }}-${{ hashFiles(format('bootstrap/linux-{0}.json', matrix.ghc)) }}-
+          key: bootstrap-${{ steps.get-osver.outputs.osver }}-${{ matrix.ghc }}-${{ hashFiles(format('bootstrap/linux-{0}.json', matrix.ghc), 'Cabal/Cabal.cabal', 'Cabal-syntax/Cabal-syntax.cabal', 'Cabal-hooks/Cabal-hooks.cabal', 'hooks-exe/hooks-exe.cabal', 'cabal-install-solver/cabal-install-solver.cabal', 'cabal-install/cabal-install.cabal') }}-${{ github.sha }}
+          restore-keys: bootstrap-${{ steps.get-osver.outputs.osver }}-${{ matrix.ghc }}-${{ hashFiles(format('bootstrap/linux-{0}.json', matrix.ghc), 'Cabal/Cabal.cabal', 'Cabal-syntax/Cabal-syntax.cabal', 'Cabal-hooks/Cabal-hooks.cabal', 'hooks-exe/hooks-exe.cabal', 'cabal-install-solver/cabal-install-solver.cabal', 'cabal-install/cabal-install.cabal') }}-