Proposal: run the full test suite on all platforms, more often

[huonw]

Currently, CI runs the full Pants test suite on Linux x86-64, via Github's free runners, but a only limited subset of them (marked platform_specific_behavior "PSB") on the other three supported platforms, if at all. This means that problems can slip through when we don't realise something is platform specific, e.g. #19940 leading to #20184 (context).

Summary of what both PR and post-merge CI does, as best I can tell:

OS	CPU	Provided by	full tests	PSB tests	Rust tests^	Wheels^
Linux	x86-64	GitHub	✅	implied	✅	✅
macOS 11	x86-64	GitHub	❌	✅	✅	❌
Linux	arm64	Self-hosted	❌	✅	✅	✅
macOS 11	arm64	Self-hosted	❌	❌	❌	✅
macOS 10.15	x86-64	Self-hosted	❌	❌	❌	✅

^Rust tests and wheel-building only runs when required.

That is, there's some platforms that are completely untested, and others that only ever have limited testing.

Proposal: Running the full test suite on the hosted machines more often, somehow.

For instance:

1. Running everything on all platforms, for every PR CI run: this almost certainly takes too much CI resources?
2. Use a merge queue that runs the full test suite on all platforms, while normal PR CI runs a reduced set (i.e. fast/low-resource-usage feedback for iteration, but follow the "not rocket science" rule for code to actually land https://graydon2.dreamwidth.org/1597.html ). Potentially normal PR CI could be reduced further since it's no longer the main gate, e.g. only run on the free GitHub-hosted runners, and use none of the self-hosted runners.
3. A semi-regular build of main and 2.*.x branches, e.g. nightly, weekly, that the MOTW checks
4. A full test run either just-before or just-after a release, as part of the release process, again checked by the MOTW.
5. Something else?

I'm personally inclined towards 2, potentially using Github's recent-ish support for merge queues: https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/incorporating-changes-from-a-pull-request/merging-a-pull-request-with-a-merge-queue

What do you think?

[thejcannon]

I'm +1 on 2, I think. And if it becomes too burdensome we can relax to 3 or 4.

One thing I'll say though is in addition to this, we really ought to work on understanding the boat we're in. What I mean by that is:

• What tests take inordinately long
• If our dependency graph can be made such that a change causes less of a ripple effect
• Understand if our caching strategy isn't working out quite as we expected

[huonw]

Just noting: with more testing on ARM macOS, we might've caught #20765 before cutting releases. (Depends on the details though, since that's sensitive to the exact arrangement of a stateful cache.)

[huonw]

GitHub now has more free public runners than it used to:

• Linux (and Windows) ARM: https://github.blog/2024-06-03-arm64-on-github-actions-powering-faster-more-efficient-build-systems/
• macOS ARM: https://docs.github.com/en/actions/using-github-hosted-runners/about-github-hosted-runners/about-github-hosted-runners#standard-github-hosted-runners-for-public-repositories

These may not be "perfect", e.g. we might want to build for older versions of macOS than 14 for releases. But they'd allow doing more good-enough testing cheaply, without over-burdening our self-hosted runners.

[cburroughs]

Use a merge queue that runs the full test suite on all platforms

It would also be nice if we built wheels in a merge queue, as that is a historical source of breakage slipping in.