Rewrite release CI

[hasufell]

This is a rewrite and an extension of #9437

The main motivation is to:

1. have a reliable release CI that runs (mostly) on unmanaged runners
2. have access to GitHubs CI infrastructure for faster and more reliable execution
3. unify the CI so that we only need to maintain one place/flavor of CI (ease of maintenance, less knowledge required for contributors etc.)

The major difference to the previous PR is that we run the "integration tests" of cabal-install (because that is the main part that diverges from the validate pipeline in terms of configuration and build flags).

[mpickering]

Perhaps it would be good to first establish what the goals with this PR is as it seems to be doing a few different things.

I think as a first step, migrating to github CI should match the configurations and artifacts produced by the existing release CI. Afterwards, once that is achieved then further platforms and packaging tweaks can be explored in a more controlled manner.

It seems risky to both migrate to a different CI platform, modify the supported platforms and also modify packaging options all in one commit. (Not that I am expressing a judgement about the value of also achieving those things as individual goals)

[hasufell]

I think as a first step, migrating to github CI should match the configurations and artifacts produced by the existing release CI

The gitlab configuration contains EOLed distros and lacks many of the artifacts that the current PR provides. Matching gitlab would be mostly a regression.

My impression is that the current release CI is not properly maintained, which is what this PR is trying to solve.

[mpickering]

Which EOL distros are you referring to exactly? It would be good to remove those certainly.

Adding new platforms for release configurations seems to be an additive task which can be achieved in follow-up discussions if cabal maintainers decide they wish to produce binaries for new platforms.

What additional artifacts are produced? How are those different from the existing release artifacts?

Just so we are clear here Julian, I agree with the premise of this patch. Perhaps if you consider the other changes as necessary/desirable as well, another way forward would be to produce a detailed list of what is the same/different to before, so each one can be considered individually during a review.

[hasufell]

Which EOL distros are you referring to exactly?

There are some updates I should do to this PR as well, but here is the inventory:

Tarball	GHCup	Cabal	Remark
x86_64-linux-deb10	-	+	EOL
x86_64-linux-deb11	+	+
x86_64-linux-deb12	+	+
x86_64-linux-fedora33	+	+	EOL
x86_64-linux-fedora36	-	+	EOL
x86_64-linux-fedora37	+	-	EOL
x86_64-linux-fedora38	-	+	EOL
x86_64-linux-rocky8	+	+
x86_64-linux-rocky9	+	-
x86_64-linux-ubuntu20_04	+	+
x86_64-linux-ubuntu22_04	+	+
x86_64-linux-ubuntu24_04	+	+
x86_64-linux-mint20_3	+	-	EOL
x86_64-linux-mint21_3	+	-
x86_64-linux-mint22_1	-	-
x86_64-linux-alpine3_12	+	+	EOL, static
x86_64-linux-alpine3_19	+	-	static
x86_64-linux-alpine3_20	-	+	static
x86_64-linux-void-glibc	+	-	rolling
x86_64-freebsd14_2	+	-
x86_64-darwin	+	+
x86_64-windows	+	+
aarch64-linux-deb10	-	+	EOL
aarch64-linux-deb11	+	+
aarch64-linux-deb12	-	+
aarch64-linux-alpine3_18	-	+	EOL, static
aarch64-linux-alpine3_21	+	-	static
aarch64-darwin	+	+
i386-linux-deb10	-	+	EOL
i386-linux-alpine3_12	-	+	EOL, static
i386-linux-alpine3_20	-	+	static
i386-linux-alpine3_21	+	-	static

[hasufell]

Our strategy going forward is as follows:

• we want to reduce the linux builds to 2:
  • one fully statically linked against musl
  • one dynamically linked against the oldest glibc that's not EOL, but statically against other libraries (gmp, ncurses etc)... this works via pointing --extra-lib-dirs to a dir which includes only static library archives (cabal only seems to link against gmp and I have a working example binary)
• then all known glibc based distros use the glibc one and alpine/unkown distros can use the static musl one

We end up with the following build environments:

Build env	Remark
x86_64-linux-rocky8	glibc
x86_64-linux-alpine3_22	musl
x86_64-freebsd14_2
x86_64-darwin
x86_64-windows
aarch64-linux-rocky8	glibc
aarch64-linux-alpine3_22	musl
aarch64-darwin
i386-linux-deb11	glibc
i386-linux-alpine3_22	musl

[mpickering]

Our strategy going forward is as follows:

* we want to reduce the linux builds to 2:
  
  * one fully statically linked against musl
  * one dynamically linked against the oldest glibc that's not EOL, but statically against other libraries (gmp, ncurses etc)... this works via pointing `--extra-lib-dirs` to a dir which includes only static library archives (cabal only seems to link against gmp and I have a working example binary)

* then all known glibc based distros use the glibc one and alpine/unkown distros can use the static musl one

This proposal seems to be a different approach to building artifacts than the cabal project has used in the past, could you open an issue to discuss this proposal with the other maintainers? It seems to be a different topic to this MR.

It would be good to build less binaries overall, perhaps this approach could be made easier by adding some features of options to cabal-install if it works out well.

[Mikolaj]

Our strategy going forward is as follows:

Indeed, your strategy seems very attractive and quite radical. Is it applicable to a different major project, such as GHC? or ghcup itself or something else major? I'd feel more comfortable if I could contant existing users of such a strategy and even more comfortable, if we could piggy-back on GHC, as we do currently, using CI very similar to what GHC uses and taking advantage of the professionalism of @chreekat and good will of the Haskell Foundation in the management of the release CI. Did GHC consider this strategy?

[hasufell]

if we could piggy-back on GHC, as we do currently, using CI very similar to what GHC uses and taking advantage of the professionalism of @chreekat and good will of the Haskell Foundation in the management of the release CI

@Mikolaj I find your response incredibly disrespectful as a contributor, because you're insinuating "we don't trust your expertise on this matter".

I'm being paid by IOG to do this type of work and we will do it anyway, with or without upstream. But as we've been asked by @bgamari and others during ZuriHac whether we intend to contribute back upstream, here is the answer: yes we do. And here is our proof. In turn, this is a great opportunity for the cabal team to demonstrate that they're willing to collaborate with other teams outside of their core "cabal".

Also as a side note: the Haskell Foundation itself apparently trusts my expertise on this matter as they are funding my proposal (and I am the person managing the Haskell Foundation GitHub runners).

That doesn't mean our goals align and I'm willing to make adjustments to this PR, but I can't work with you, unless the commentary remains professional and technical.

[Mikolaj]

Edited: I don't feel I can productively contribute to reviewing this PR, so let me thank @mpickering for discussing its merits up to this point and let ma ask @mpickering and other cabal maintainers and developers to carry on.

[mpickering]

Right, I separately chatted with Julian and Mikolaj. It does seem like there are some unresolved tensions from past interactions. Thank you for your gracious reflection Mikolaj that you would find engaging with this PR difficult.

With @hasufell I discussed the ways that we could move this PR forward, and that we should agree a shared plan of work.

Julian is quite keen to make the CI line-up more with what's required for ghcup, which is understandable since ghcup is the primary way which cabal binaries are distributed.

Cabal maintainers are keen to not have a system which relies on one person (Julian) to maintain, and wish to understand the new system carefully so they can use it without external help in future.

• Julian said to me what as a first step he could make the platforms in this PR align more, that seems like an uncontroversial way to make progress. But if maintaining platform compatibility is difficult for other reasons, we are not opposed to also updating distributed platforms at the same time.
• Matt can then take a careful look at the patch, with the mindful goal of something which can be maintained without Julian's assistance if necessary.
• The new approach to making binaries sounds interesting, but should be discussed on a ticket and via an experimental job before committing to go down that track for the actual releases. Julian will make a ticket where this can be discussed more broadly.

Does that match your impression from our conversation @hasufell ?

[hasufell]

@mpickering that sounds resonable.

Tomorrow I will start:

• splitting the PR into smaller chunks (e.g. the platform updates in a separate commit)
• moving the release stuff to a reusable workflow, so it can be triggered from other places
• open a ticket about reducing linux builds and how to achieve it

with the mindful goal of something which can be maintained without Julian's assistance if necessary.

Well, I think that the bus factor will be much better with the move to GitHub, since most cabal contributors already have exposure to GitHub CI and it doesn't require intimate knowledge of runners (except for FreeBSD, but I'm confident that @geekosaur also has the required expertise to handle it).

[ffaf1]

From CONTRIBUTING.md.

A pull request fixes a problem that is described in an issue. Make sure to file an issue before opening a pull request. In the issue you can illustrate your proposed design, UX considerations, tradeoffs etc. and work them out with other contributors. The PR itself is for implementation.

I would too like to ask for an issue with goals clearly stated for sure, and possibly trade-offs and resources considerations.

[hasufell]

@ffaf1 I can do that, but please note that the motivation and merits have been discussed for years and I would like to assume by now Cabal developers are well aware. There even have been synchronous calls about this topic and multiple Cabal developers/contributors have expressed positive interest in this approach.

So yes, I can open an issue with motivation and explanation, but I won't be participating in argumentative discussion, nor will I try to prove why GitHub is a better decision for release CI in the context of Cabal. This is up to you to decide.

This is work that we're using over at stable-haskell and I'm happy to upstream it and invest additional effort into making it upstreamable.

[hasufell]

#11078

[mpickering]

Thanks for making the ticket Julian, that's just a normal part of contributing to cabal. Having a place to discuss the issue which isn't on the PR is useful for the longevity of recording these discussions.

I'll get back to looking at this next week when I'm back at work if you have made the changes we discussed.

[hasufell]

Only thing left in this PR is executing on the "only 2 bindists for linux" plan in a separate commit.

[hasufell]

Yes

[geekosaur]

Ugh. GitHub was hiding the unresolved comment because it was on a commit that went away during a force-push. I finally found and resolved it.

[mpickering]

Thanks everyone who contributed to this. I'm glad we've managed to consolidate onto GitHub eventually. I do think that will make things easier going forward.

Before we all move on, I think it's worth reflecting on the discussion in this thread. A lot of the tension here wasn’t really about technical decisions, it was more about the social factors of developing software together.

It seemed to me that there are sometimes mismatched assumptions about how to balance individual initiative with collective ownership and processes. That mismatch can lead to frustration on both sides: people feeling blocked or untrusted, and others feeling bypassed or sidelined.

This thread took quite a bit of emotional energy to manage, and I don’t plan to take on that role again by default. I think we could get better at recognising when a conversation has shifted from technical disagreement into something more social, and try to address that openly rather than letting it smoulder in the background. To me, it undeniable that a certain tension exists and that we would be better off addressing that openly so that working together is smoother in future.

[geekosaur]

I have a feeling none of us is good at managing social issues. Certainly I'm not, and in the past I've wrecked myself in various ways by trying (my brain overloads quickly in some ways). This may mean finding someone with more social chops to be a manager.

[mpickering]

I have a feeling none of us is good at managing social issues. Certainly I'm not, and in the past I've wrecked myself in various ways by trying (my brain overloads quickly in some ways). This may mean finding someone with more social chops to be a manager.

Thanks Brandon, that seems like a useful thing to realise anyway. Everyone has their strengths and ways to contribute to the project.

[hasufell]

@mpickering thanks for your perspective.

I have no authority over the cabal maintenance process, but I can shed some light why I felt this contribution to cabal was extremely cumbersome. The only reason I attempted to upstream this (again) at all was because I got paid by IOG to do so. I want to make this very clear: I would never have contributed this in my free time. Maybe that is something cabal developers should be concerned about.

Now, this isn't the first time this change was proposed and in previous discussions certain cabal developers made claims that were outright wrong (something along the lines of "there's something special about GHC CI and cabal would do better to produce the release binaries there for correctness reasons"). I pointed out that this is simply not true and backed it up by demonstrating so by producing cabal binaries downstream. This was not enough and those developers continued to block this change, while others expressed interest in my approach.

We should be able in a tech community to point out incorrect assumptions and cherish truth seeking. This is more so important because we are a too diverse group of people and cannot rely on merely social trust or liking each other to drive contributions.

Note that I'm not complaining about the decision back then, but the reasoning that I was able to prove wrong. And some interactions during this contribution have reinforced my belief that those were indeed emotional reasons. It felt like negative personal bias and I think that has no place in the Haskell community.

I'm happy that we got past this somehow, but I continue to be skeptical about the cabal contribution workflow until I see that there is a strong commitment to

• good engineering
• open collaboration
• truth seeking

I don't think the ownership model (BDFL, etc.) makes any difference... as long as those qualities are exercised.

[mpickering]

Thanks for your perspective as well, @hasufell. It does say a lot that this experience has left you feeling completely put off from contributing to cabal development. You are very knowledgeable and passionate about the Haskell ecosystem so that is a shame. It also sounds like it took a lot of personal effort for you to participate in this process at all.

I’d like to understand more about how it’s felt from your side. From what you’ve said, it seems frustrating that the evidence you provided didn’t feel sufficient to move the discussion forward. What was it like to revisit the same topic again after the earlier experience?

I’d also like to invite Cabal maintainers to share their own personal reflections, not about who was right or wrong, but about what it was like for them to engage in the discussions on this PR and previous ones. My hope is that we can speak openly about how these interactions have affected us personally, so we can learn from that before getting into technical reasoning about how to improve the process.

From my perspective, we did manage to keep the discussion on-topic for this contribution and arrive at a PR that fits with existing Cabal practices while also including your ideas for improvement. That’s a positive outcome and I think it’s even more positive that we’re now also looking at the social side of collaboration.

[hasufell]

From what you’ve said, it seems frustrating that the evidence you provided didn’t feel sufficient to move the discussion forward. What was it like to revisit the same topic again after the earlier experience?

I was worried about dealing with irrational arguments.

Those are the most exhausting to deal with and that has very little to do with the actual decision.

That's why I continue to believe that the way it was handled in the past was improper conduct. So I'd be interested to see if the cabal team is planning to express similar values as these: https://github.com/haskell/core-libraries-committee?tab=readme-ov-file#our-values

[Mikolaj]

@hasufell, @mpickering, @geekosaur and everybody, congratulations for landing such an important and well-considered PR.

It seemed to me that there are sometimes mismatched assumptions about how to balance individual initiative with collective ownership and processes.

I'd be grateful if somebody could point out to me how mismatched assumptions (or maybe some kind of miscommunication on my part?) led to what upset me in the conversation in this PR to the point that I excused myself from contributing to it. Namely, that Julian said in #11072 (comment) "I can't work with you, unless the commentary remains professional and technical" and then announced out of the blue in the public cabal room on Matrix "Man of Letters: can you please stay professional on the ticket?" in response my comment:

"

Our strategy going forward is as follows:

Indeed, your strategy seems very attractive and quite radical. Is it applicable to a different major project, such as GHC? or ghcup itself or something else major? I'd feel more comfortable if I could contant existing users of such a strategy and even more comfortable, if we could piggy-back on GHC, as we do currently, using CI very similar to what GHC uses and taking advantage of the professionalism of chreekat and good will of the Haskell Foundation in the management of the release CI. Did GHC consider this strategy?
"

This was my only public comment about this PR up to that point. The "strategy" in the line cited from Julian's earlier comment was to do with "reduce the linux builds to 2" and it's eventually not implemented in the merged PR AFAIU, so I guess this point is not resolved still.

[hasufell]

and then announced out of the blue in the public cabal room on Matrix "Man of Letters: can you please stay professional on the ticket?" in response my comment:

This is not true. You deleted the comment that caused me to go on Matrix and call you out on your behavior.

[Mikolaj]

I apologise. You are right, my disengagement comment (the edit history shows the first version as "Julian, you are starting a drama and, this time, I'm not taking the bait." and the current version is the second version from the next day, saying "Edited: I don't feel I can productively contribute to reviewing this PR, so let me thank @mpickering for discussing its merits up to this point and let ma ask @mpickering and other cabal maintainers and developers to carry on.") was in-between your github comment and the Matrix comment.

My question stands.

[mpickering]

I think in this discussion we are getting quite focused on the idea of someone being right and wrong. People can experience the same interaction differently without one being "the truth" or "false".

It's more useful if we can talk about things which we certainly know to be true, how the situation felt to us and affected oneself. It's very useful that you've done that Julian and shared your experience. "Exhausting" is a strong word to use when talking about these issues and definitely gives a sense about what kind of problem these interactions cause you to feel.

Julian, you’ve also described some past reasoning as "irrational" which I can see would be frustrating when you’ve brought evidence. If we presume good faith, what do you think might lead someone to make an argument like that? Understanding this might make it easier to avoid or resolve such situations more quickly in future. It's hard for everyone to work from exactly the same playbook.

Mikolaj, I also get the sense that you felt hurt by these exchanges. It’s possible that earlier disagreements around this PR contributed to a more charged atmosphere, and that your comment was interpreted in light of that history rather than on its own terms.

To me, it doesn't seem like a great situation to be in when these discussions leave everyone involved feeling deterred. That's why I'm interested in helping to resolve this situation. It seems that everyone will be better off if these situations are less tense in the future.

It seems to me that if cabal maintainers were to write about the norms expected from communication then they would propose something similar to what you have suggested for the CLC. I think many groups operate under these kinds of social expectations. What do you think of Julian's document @Mikolaj?

[hasufell]

It's more useful if we can talk about things which we certainly know to be true

Yes. Let's make this clear: There is nothing special about GitLab CI. Cabal does not need to be built "close to GHC". There are no technical reasons for this.

This claim ("matching GHC environment") has been repeated over and over again here: #8674 (comment)

With the final argument being "This is gut feeling and guesswork all the way.".

This was a display of authority showing "I don't need a good argument to reject your proposal". The following interactions with Mikolaij (including in this thread) also showed incredible personal bias, being dismissive in nature and totally void of any substantial insight.

I don't believe that this is how any project part of the Haskell core tooling should operate. If you lack expertise somewhere (as do I... I barely know the cabal codebase), ask people who have it. You can still make your own decision after you understand the relevant technical facts.

[geekosaur]

That's part of the problem: the people who have it are gone. We're all flying by the seats of our pants, and trying to understand why some things were done the way they were.

[hasufell]

@geekosaur then it might be a good idea to not alienate contributors who bring in expertise.

[Bodigrim]

That's part of the problem: the people who have it are gone. We're all flying by the seats of our pants, and trying to understand why some things were done the way they were.

Being an infrequent contributor but a frequent lurker, my gut feeling is that the Cabal team is sometimes prone to make up for the lack of expertise by erring on the side of caution. I'm saying "prone", because one cannot perform a proper risk assessment without actually having an expertise. Doing so nevertheless creates a frustrating environment for contributors: if you ask me to prove that something does not break unknown unknowns, I am stuck, because there is likely no constructive proof of such thing.

The usual recipe in such scenario is extending automated quality assurance in various directions until reaching a point at which you feel confident to merge pretty much anything which passes tests. Say, building head.hackage or clc-stackage with Cabal HEAD would be an easily accessible step in this direction.

[geekosaur]

head.hackage is probably a lose unless building ghc HEAD. clc-stackage is a possibility, but cron stuff has annoyances like only sending notifications to the last person who committed to them. That said, I've been working toward things like that with #10503.

[Bodigrim]

I think head.hackage is kept compatible with older GHC releases as well. But my point was specifically testing both GHC HEAD and Cabal HEAD as early as possible so that we can detect problems early. (Shall we move this more specific discussion elsewhere?)

[hasufell]

The usual recipe in such scenario is extending automated quality assurance in various directions

Well, my PR that reduces release binaries runs the installation on all major distros and executes the cabal-install integration test.

Additionally, I've shipped the last cabal release with exactly that strategy in GHCup and have not received any bug reports since then.

We can safely assume these binaries are now used all over github CI.

And yet, that wasn't enough work to convince cabal maintainers, as Matthew indicated he wants things to "settle" first.

So I'm not sure it's a question about testing.