Split excluded_pkgs into separate groups

Normally, each package in a given package set is used in two ways:

  - It is written to generated.cabal's build-depends, hence built.

  - Its exact version is written to cabal.project.local as a
    constraint, for reproducibility of transitive dependencies.

We used excluded_pkgs.jsonc to opt out of one or both of these for a
given package e.g.

  - Totally exclude cuda from the build.

  - Exclude boot libs like text from having their exact versions
    written, as these libs will be used transitively anyway, and exact
    pins complicate building with multiple GHCs.

We since stumbled onto a third scenario: packages that we do not want
in build-depends yet benefit from being pinned. For example,
we do not want happy in build-depends (since it is an exe), yet we
would like to use stackage's pins, so that cabal does not choose a
version that might not work.

We therefore modify excluded_pkgs.jsonc to package_index.jsonc, which
separates these categories into different json keys:

  - excluded: Same as before, these packages are filtered out from the
    given package set.

  - excluded_pinned: Packages that are filtered from the package set
    yet we want their pins written anyway e.g. happy.

  - unpinned: Packages to be built but should not be pinned e.g.
    boot libs.

Author: tbidne

README.md

@@ -76,7 +76,7 @@ Because we build with `nightly` and are at the mercy of cabal's constraint solve
 
   - `p` requires a new system dependency (e.g. a C library).
   - `p` is an executable.
-  - `p` depends on a package in [./excluded_pkgs.jsonc](excluded_pkgs.jsonc).
+  - `p` depends on an excluded package in [./package_index.jsonc](package_index.jsonc).
 
 - A cabal flag is set in a way that breaks the build. For example, our snapshot requires that the `bson` library does *not* have its `_old-network` flag set, as this will cause a build error with our version of `network`. This flag is automatic, so we have to force it in `generated/cabal.project` with `constraints: bson -_old-network`.
 
@@ -95,17 +95,17 @@ We attempt to mitigate such issues by:
 
 Nevertheless, it is still possible for issues to slip through. When a package `p` fails to build for some reason, we should first:
 
-- Verify that `p` is not in `excluded_pkgs.jsonc`. If it is, nightly probably pulled in some new reverse-dependency `q` that should be added to `excluded_pkgs.jsonc`.
+- Verify that `p` is not in `package_index.excluded`. If it is, nightly probably pulled in some new reverse-dependency `q` that should be added to `package_index.excluded`.
 
 - Verify that `p` does not have cabal flags that can affect dependencies / API.
 
 - Verify that `p`'s version matches what it is in the current snapshot (e.g. `https://www.stackage.org/nightly`). If it does not, either a package needs to be excluded or constraints need to be added.
 
 In general, user mitigations for solver / build problems include:
 
-- Adding `p` to `excluded_pkgs.jsonc`. Note that `p` will still be built if it is a (transitive) dependency of some other package in the snapshot, but will not have its exact bounds written to `cabal.project.local`.
+- Adding `p` to `package_index.excluded`. Note that `p` will still be built if it is a (transitive) dependency of some other package in the snapshot, but will not have its exact bounds written to `cabal.project.local`.
 
-- Manually downloading a snapshot (e.g. `https://www.stackage.org/nightly/cabal.config`), changing / removing the offending package(s), and supplying the file with the `--snapshot-path` param. Like `excluded_pkgs.jsonc`, take care that the problematic package is not a (transitive) dependency of something in the snapshot.
+- Manually downloading a snapshot (e.g. `https://www.stackage.org/nightly/cabal.config`), changing / removing the offending package(s), and supplying the file with the `--snapshot-path` param. Like `package_index.jsonc`, take care that the problematic package is not a (transitive) dependency of something in the snapshot.
 
 - Adding constraints to `generated/cabal.project` e.g. flags or version constraints like `constraints: filepath > 1.5`.
 

dev.md

@@ -20,7 +20,7 @@ The `clc-stackage` library is namespaced by functionality:
 
 ### parser
 
-`CLC.Stackage.Parser` contains the parsing functionality. In particular, `parser` is responsible for querying stackage's REST endpoint and retrieving the package set. That package set is then filtered according to [excluded_pkgs.json](excluded_pkgs.json). The primary function is:
+`CLC.Stackage.Parser` contains the parsing functionality. In particular, `parser` is responsible for querying stackage's REST endpoint and retrieving the package set. That package set is then filtered according to [package_index.jsonc](package_index.jsonc). The primary function is:
 
 ```haskell
 -- CLC.Stackage.Parser
@@ -77,7 +77,7 @@ The executable that actually runs. This is a very thin wrapper over `runner`, wh
 
 `clc-stackage` is based on `nightly` -- which changes automatically -- meaning we do not necessarily have to do anything when a new (minor) snapshot is released. On the other hand, *major* snapshot updates will almost certainly bring in new packages that need to be excluded, so there are some general "update steps" we will want to take:
 
-1. Modify [excluded_pkgs.json](excluded_pkgs.json) as needed. That is, updating the snapshot major version will probably bring in some new packages that we do not want. The update process is essentially trial-and-error i.e. run `clc-stackage` as normal, and later add any failing packages that should be excluded.
+1. Modify [package_index.jsonc](package_index.jsonc) as needed. That is, updating the snapshot major version will probably bring in some new packages that we do not want. The update process is essentially trial-and-error i.e. run `clc-stackage` as normal, and later add any failing packages to `package_index.excluded` that should be excluded.
 
 2. Update `ghc-version` in [.github/workflows/ci.yaml](.github/workflows/ci.yaml).
 
@@ -112,3 +112,7 @@ $ NO_CLEANUP=1 cabal test functional
 ```
 
 Note that this only saves files from the _last_ test, so if you want to examine test output for a particular test, you need to run only that test.
+
+> [!TIP]
+>
+> CI has a job `build-batch` which actually builds the entire package set, hence it can be used in place of manual building / testing. Note it takes about an hour to run.