Improving Post-Merge Stability -- How Can We Help?

[drhodrid]

Dear Firedrake Developers,

First, I just want to say how much we appreciate the effort that goes into Firedrake’s continued development. The work being done is invaluable, and we completely understand the need for recent merges to the master branch. I’m reaching out not as a complaint, but because we’d really like to contribute to making the development process smoother for everyone.

Over the past few months, we’ve hit a few major issues after merges, typically with larger parallel jobs - but not always. Debugging and fixing these has often taken quite a bit of time, sometimes involving people who aren’t necessarily experts in those parts of the code. From our side, Angus, Sia, Dale, and others have been working through issues, raising PRs, and engaging directly with developers to help resolve them. While we’ve managed to work through things, we all want to see Firedrake continue to grow and thrive, and we’d love to help reduce the frequency of these post-merge problems.

This thread is therefore intended only to start a discussion around how best to facilitate this. For example, would it be possible to expand the test suite to catch more of these issues before merges happen? The move to incorporate Gusto tests was great, and we’d love to do the same with G-ADOPT (Angus has already chatted with Connor about this on Slack). That said, the current 3-minute, 12-core test limit doesn’t allow us to check many critical aspects of the framework. If computational resources are a constraint, we’d be more than happy to help with testing - whether that means running tests on a cluster, contributing to GitHub Actions, or finding another way to support broader test coverage.

The ideal scenario would involve every Firedrake merge passing the G-ADOPT (and gusto etc?) CIs... Is this, or something similar, possible?

We’d really like to work together on this to improve the process and minimise these post-merge surprises. Looking forward to any thoughts or suggestions on how we can contribute to making things more robust!

Best wishes,
Rhodri

[drhodrid]

This is closely related to #4008, albeit with a different focus, hence the different discussion item.

[dham]

I can't see how we could feasibly run your and all the other downstream CIs on every merge. This would generate an intricate and fragile dependence on test infrastructure that we don't control and downstream changes that we don't understand.

We are putting a lot of work at the moment into versioned releases as a mechanism for providing more Firedrake stability to downstream packages. This is the standard way of addressing this issue and I think the better way to deal with the issue.

[connorjward]

The 3 minute, 12 core limit was only a suggestion. We are hoping to test a number of upstream packages and don't want things to balloon out of control. If you want a little more CI time that's no issue.

We are hoping to introduce versions releases (#4013) soon which should really help with the breaking changes issues.

[sghelichkhani]

Thanks, @dham and @connorjward , for your responses. The move toward versioned releases is a great step that could improve stability for downstream projects. Its effectiveness though depends on ensuring that releases don’t introduce breaking changes in dependent packages.

Would there be any interest in exploring something like a repository_dispatch event, triggered only on the release branch (not the development branch), to verify that all downstream packages pass their tests before a version is finalised? Without some mechanism like this, a release version that still causes issues in downstream projects may not fully address the problem.

That said, this structured release approach also gives us the opportunity to align with regular release cycles, which could be useful.

[connorjward]

Hi @sghelichkhani, I think we do not intend on making many changes to our release branch. I envisage us only making changes when, say, an upstream dependency makes a breaking release and we need to pin to an existing one, or if macOS make a breaking compiler update and we need to update the PETSc version we depend on. Therefore doing a mammoth CI run before merging seems like overkill.

[drhodrid]

Thanks, everyone, for your input. Moving forward, we’ll aim to ensure that the G-ADOPT smoke tests on Firedrake cover as many critical areas as possible, building on @angus-g’s recent work: #4080. As Sia suggested, aligning our testing with Firedrake’s release cycles seems sensible. Hopefully, our expanding local CI - which already includes several HPC-forward and adjoint tests - will help us catch and address any potential bugs or issues in a timely manner.