Use macos-26 runners and test on Intel/ARM

[sirosen]

Open as a draft because I think this needs to be linked to relevant issues. It also needs to actually run in CI to confirm it works as desired.

• In CI, separate OS name from runner version
• Update CI to test both Intel and ARM macOS

Contributor checklist

• Included tests for the changes.
• A change note is created in changelog.d/ (see changelog.d/README.md
  for instructions) or the PR text says "no changelog needed".

Maintainer checklist

• If no changelog is needed, apply the bot:chronographer:skip label.
• Assign the PR to an existing or new milestone for the target version
  (following Semantic Versioning).

[sirosen]

This appears to be working as desired, but I find the results in CI wall clock time a bit concerning.
CI time in this PR seems to be dominated by macOS runs, and they're taking ~10 minutes (or 100%) more time to complete than previous CI runs, e.g. on main.

I'm also worried that by increasing the number of builds by so much, we're increasing the changes of flaky failures. We want to solve those anyway, but "want to fix" != "have fixed".

Many projects consider it sufficient to test their max and min Python versions on Windows and macOS. I think I would be comfortable doing that at least for these new Intel-mac builds, to keep CI times more under control. It would require refactoring the job config a bit, but I think it's necessary.

[webknjaz]

I had an idea some time ago inspired by what CPython does with buildbots on exotic OS/hardware variants but haven't tried it yet. What they do is they run testing at some point post-merge due to resource availability. And if something fails, PRs get a comment with a link to the failure. The authors are supposed to fix it within 24 hours or a PR gets reverted.

The gist of what I'm considering is that we have a separate matrix for slow jobs that isn't tied into alls-green. That matrix has a special GitHub Environment assigned (slow-jobs or whatever), similar to the publishing job. But the difference would be that this env would have a cooldown timer as the only protection rule enabled. With this, the usual jobs would run quick and get a responsive status update. And if one would be pushing more PR updates, their older commits will only have results for the quick jobs (as the slow ones would be auto-canceled). But once things settle and that cooldown timer completes (in like 20 minutes or even an hour), the PR would get the slow jobs unsuspended and they'll eventually produce some results.

One corner case to think of is merge queues as for those I imagine we wouldn't want the timer.

Would you like to experiment with this idea here? I've been thinking about it for quite a while in the context of projects with C-extensions that would be noticeably slower.

[webknjaz]

Is macos 15 slower or faster, by the way?

[sirosen]

It's hard to be confident, but so far it looks like

ARM macos 26
is faster than
(Intel) macos 15
is faster than
Intel macos 26

The main issue with CI time is that with a large number of macOS runners, we get a lot of queuing in GitHub. None of the jobs in our matrix are particularly slow -- I'm actually very happy with how fast they all are.

I'm game to experiment with new CI strategies here, but I don't want to complicate this "simple" PR with those changes. ... I think I'd like to merge this as-is, accepting that it raises the overall CI time to ~20 minutes. With the benefit of more time to think about it, that seems fine to me, since CI time isn't a major bottleneck for us. And i think we both have ideas about CI refinements that will make this better over time.

I'm marking this PR "ready" without further changes. Yes, things will get a little slower for a while. But we'll make them faster over time.

[webknjaz]

We should be using kebab-case keys for GHA to follow the convention consistently.

Also, complex non-primitive values in the matrix tend to be difficult to work with (and didn't use to be supported officially in the past). I wouldn't mind rendering the VM identifiers as is, w/o any extra fancy naming.

[sirosen]

Yep, I can make those adjustments, along with the other edits.

I also don't mind rendering the identifiers verbatim in our job names -- the previous config seemed carefully arranged to produce short names (but in, IMO, a fragile way).

[webknjaz]

Let's just turn this (and the matrix) into a simple

Suggested change

	VM-${{ matrix.os.name }},
	VM-${{ matrix.runner-vm }},

so it matches what I have in other repos as it's easier to sync.

[webknjaz]

Suggested change

	os:
	runner-vm:

[webknjaz]

Suggested change

	- name: Ubuntu
	runs_on: ubuntu-latest
	- name: Windows
	runs_on: windows-latest
	- name: macOS
	runs_on: macos-26
	- name: "macOS (Intel)"
	runs_on: macos-26-intel
	- ubuntu-latest
	- windows-latest
	- macos-26
	- macos-26-intel

[webknjaz]

(though, I'd prefer getting rid of latest for good in favor of more predicatability and tracability when looking at old CI runs)

[sirosen]

100%!

I plan to come back with a dedicated PR to simply swap latest for the appropriate version numbers. 😁