Allow hotfix releases in the pipeline

[skrech]

Extended the CI to support "hotfix" releases.

Here is the explanation of how it works:

Overall, the process is the same as before -- to create a release we need to modify VERSION file and merge it, which would trigger the release.yaml workflow. However, there would be two branches now -- main (with the meaning of development branch) and release (branch where we keep all the released stuff). release branch would be auto-created if a specific repository doesn't have it already.

So, when we modify VERSION on main, this would create a "cross-merge" between main and release branches.
This means that:

• main is merged into release, so release would get all the latest developments. Merge conflicts are auto-resolved in mains favor.
• then, release is merged back into main. This is actually a fast-forward merge (since we've just integrated all of main into release in the previous step).
• In this scenario the commit that is tagged with the released version is the merge commit and both main and release are pointing to it, at that moment.

This workflow is most likely used for releasing major changes.

The other case is when modifying VERSION on release branch, this is the case for "hotfix" releases. In this scenario:

• Before modifying the VERSION file, we expect to merge couple of PRs with cherry-picked changes (from main) into release.
• On VERSION file changed, the CI would update the CHANGELOG (would contain only changes that are in the PRs targeted at release), and tag that commit with the new version tag.
• Only release branch is merged into main. This would allow for git describe from main to track how many commits are done since last releasing. The merge auto-resolves conflicts, again, in mains favor. Please note, the tag for this release is made on a commit that is not the merge commit, but rather the one before it, made on the release branch (this is the changelog-updating commit).

NOTES:

• We're working with the assumption that every change is first put on main and then "cherry-picked" onto release. The same approach as OBS's factory-first policy.
• CHANGELOG-updating commit is always made on the branch that triggered the release.
• If this PR gets approved, I'll move git-release, obs-sync and publish-containers workflows into trento-project/.github to make them common re-usable blocks for all the rest of the trento projects. Following that, I'll remove them from this repo.

I've used my personal fork to try this out, and you can see how git log --graph looks like here:
https://github.com/skrech/ansible

Changelog handling caveat

Although this branching strategy looks sound and integrates well with container builds/publishes and obs sync, there is a slight problem with CHANGELOG handling. To understand the problem I need to describe how the changelog handling is done in our repos. It's based on a github action called release-drafter and works like the following:
1a. release-drafter gets all the releases in a repo (using the GH API);
2a. sorts them in an ascending order and gets the latest one;
3a. creates an entry for every Pull Requests targeted at the branch that would be the base for the new release;
4a. the PRs considered are these that are made in the time window between discovered latest release (in 2.) and the top of the target branch.

This GH actions, thus, implies two constraints:
1b. Every change in a changelog should have a corresponding PR, the action can't extract changelog entries from commits only.
2b. release-drafter action only considers very simple use-cases where releases are happening in strictly chronological order and on a same branch.

This effectively means that if we make hotfix release and there is work done on main already, on the subsequent release of main, changes happened before the hotfix won't have changelog entries. This is obviously, non-acceptable.

To solve this problem, I'll tackle the constraints enumerated above in reverse order:

2b. In my opinion release-drafter has very limited support for various git branching strategies. It uses the GH graphql API to get the commits with their associated PRs using since history lookup. However, we have the case where two branches share a common base commit but at the time of release they contain different commits. So, when we make release 4.0.0 but there were already two hotfix releaes 3.0.1 and 3.0.2, we want to include all the changes on main since 3.0.0 but not the changes introduced in 3.0.1 and 3.0.2. This is trivial to express in git: git log 3.0.2..4.0.0. But it's not how release-drafter works.

So, a workaround is introduced by me to handle that: release-drafter supports a config options called filter-by-commitish, which takes part in step 2a -- in addition to sorting the releases it also filters them by the value of field target_commitish, eg. by the branch on which the release has been made. So, when making full releases, we consider only version 4.0.0, 3.0.0, and so on. When making hotfix releases we consider every release 3.0.2, 3.0.1, 3.0.0, etc.

This has implications that we have to pay attention to tag the full release on main branch even though both branch heads main and release point to the same commit. target_commitish is GH API fields only. Hotfix releases are tagged on release branch as expected, no special attention needed for them.

Unfortunately, filter-by-commitish is only available via release-drafter.yaml, so I had to make two version of this config file, one for full releases and one for hotfix releases, depending on which branch is triggering the release. These files are also needed in connection to 1b. as well.

1b. As already mentioned, release-drafter includes only PRs in the changelog, not commits. This is actually not a big deal for us but it implies we create PRs to release branch when introducing the hotfix changes. This is a bit clumsy because for a hotfix branch, one usually uses cherry-pick operation.

There is more to this though, since this is not a cherry-pick operation, there is no way for git/gh to understand that a hotfix change was already released as part of hotfix release. Thus, there would be duplicate changelog entries when releasing full releases. To solve this, we have to mark every change from main that is released in a hotfix with newly-introduced released-as-hotfix label. When this label is set, release-drafter would skip inclusion of that PRs message as changelog entry for the release.

Again, because release-drafter doesn't support the equivalent of git log a..b (or git log a..b --cherry-pick), it would consider both changes on main and release branch when releasing full releases. Thus, PRs on both branches should be marked as released-as-hotfix.

Anyway, don't despair. This is a lot of text to explain how things work and what were the decisions made. The take-aways are quite simple, after all:

1. Nothing changes if we're going to release only full releases, as before.
2. If you need to release a hotfix, you need to introduce the hotfix change with a PR marked with released-as-hotfix and also find the equivalent PR on main and retro-actively mark it with the same label.
3. When making full release after hotfix, you don't need to do anything.

[vicenteqa]

hey @skrech overall the strategy looks solid :)
I just added a couple of comments, also I understand that we'll need to protect release branch via repo rulesets like it's done with main branch.

[antgamdia]

comment: totally out of scope, probably, but I wonder why we aren't signing the commits. See here how we are mixing up verified and non-verified commits: https://github.com/trento-project/ansible/commits/1.0.0

I guess this is simply because we don't have it elsewhere in the project... but perhaps it could be a good idea since we are revisiting some flows.

[skrech]

Thank you for the reviews, guys. :)
@vicenteqa very good catches! Will fix them right away.
@antgamdia, yes, I noticed that too. It's because we're using shapbot name for the commits but that is not tied to a GPG key. This should be relatively easy to do -- I have to setup GPG keys for shapbot in GH and then modify the setup in the workflow.

[skrech]

Sorry, I have to re-request reviews, because of CHANGELOG handling work that might raise some concerns. PR info is updated with new section for CHANGELOG handling behaviour.

[antgamdia]

comment: Not sure if it's worth it, but... what if we check if this merge has any conflicts in advance? Something like a dry-run git merge --no-commit --no-ff origin/release; git status; git merge --abort
Perhaps it's too much, we may not need it, but wanted to mention it just in case.

[skrech]

If there are any conflicts, I think I'd like them to be auto-resolved if possible.

My thought: If we're only back-porting fixes from main into release, then merging back release into main should be clean operation -- every change into release should be already in main.
-X ours is handling the case where for example I made a change A on main, then backport it into release and before releasing I made change B on the same place in main in the context of a bigger change that should be release yet. Then, when I actually release the hotfix, I don't want that to revert change B but rather preserve it.

[antgamdia]

The take-aways are quite simple, after all:

1. Nothing changes if we're going to release only full releases, as before.
2. If you need to release a hotfix, you need to introduce the hotfix change with a PR marked with released-as-hotfix and also find the equivalent PR on main and retro-actively mark it with the same label.
3. When making full release after hotfix, you don't need to do anything.

comment: Thank you for the explanation and summary. I'm not used to the release workflows in this project, so just dropping my two cents:

Relying on a manual two-step process seems really error-prone. At the very least, we would need a piece of documentation with step-by-step instructions on how to do it.

Don't know if it is an option, but, what about creating a kind of "backport PR workflow"?

For example, assuming we have an existing PR (against main) for a hotfix, a maintainer could just comment with /backport. This way, the bot could:

1. cherry-pick the merged PR onto release and open a backport PR.
2. apply released-as-hotfix to both the backport PR and the original PR automatically.
3. cross-link the PRs in body/comments.

This way, we keep the release label-based dedupe logic, but we remove the manual work. WDYT?

[skrech]

This actually sound amazing!

But I guess this would need developing a bot. I'm actually eager to work on that if others also thing it's worth it!

[antgamdia]

Thanks :P
Not really, I meant the existing shapbot (like our hakube-bot) for the commits as the current PR does, but just using a workflow with a "comment" trigger that simply detects the / command.

Alternatively, we could even avoid the GHA action for the changelog, and build manually using external tools, like Goreleaser changelog or github-changelog-generator

[skrech]

Ah, issue_comment trigger would be piece of cake! Thanks, @antgamdia !

I really don't like this gh action but our current release was relying on it and I wanted as little changes as possible. Anyway, it works with PRs, not commits, which gives us easy grouping based on PR labels. Of course, there could be other variants relying on naming or commit labeling, but we'll have to explore them in the future. This would also allow us to potentially de-couple from GH and make the process more universal.

/backport command seems to be a quick win, for now. Then we can expand on switching components further.

[arbulu89]

Hey @skrech ,
Really appreciated the long description about the process and the issue to solve.

Without having been thinking that much about it, I was wondering why would we do

1. Fix issue in main
2. Cherry pick to release
3. Merge release back to main (maybe fixing conflicts)

Instead of simply:

1. Fix issue in release
2. Merge release back to main (maybe fixing conflicts)

At the end, the issue we want to fix is certainly present in release branch, while main could have had already some ongoing changes in the affected piece of code. In this case, backporting the fix looks maybe harder, as it cannot be cherry picked without resolving issues (this could happen in the other way as well). I find it more natural to work in the code that has exactly the "bug".
Other scenario or example could be as well that the "bug" in main was already solved during a new feature implementation, so cherry-picking a commit again is not feasible without including new features.

[skrech]

Hey @arbulu89,
There is actually no hard requirement for the change to be a cherry-pick. We can directly introduce the fix on release and then merge that into main. The merging logic should auto-resolve the conflict on main, if any.

However, if possible, it's preferable (more cleaner maybe) to fix everything on main, then back-port/cherry-pick the fix in a hotfix. Of course, if that couldn't happen, then directly crafting a fix on release is the way to go.

[skrech]

Added documentation for the process in this PR:
trento-project/docs#153

I consider this PR as complete in terms of concept. I intend to open new PRs for the good ideas mentioned in the comments:

• Signed commits
• /backport PR comment dommand for convenience
• Extracting the common workflows into project-trento/.github

Let me know if you wouldn't want to merge this without adding some of these items in this PR.

[antgamdia]

LGTM; deferring the actual approval to someone more experienced in the release process here. But for now, it is an improvement from what we had, +1 to go ahead and add improvements next. Thanks for editing the docs!

[arbulu89]

Hey @skrech ,
I think we can move forward and created new PRs if needed with improvements, to not overload the current work.

Nicely done, we just need to see it in action!

[gagandeepb]

Question: Something not very clear to me: how would we handle hotfixes to old major versions. For example, if current major version is 3.x.y and we need to support version 2.a.b for some users/deployers for whom there is no/limited upgrade path due to some reason to do a major version update for the near term.
Comment: Additionally, it would be interesting (in the future, not in this PR) to have a matrix of trento components' cross compatibility across different released versions. (not sure if this is even automate-able)

[skrech]

Hey @gagandeepb, this is in the docs: trento-project/docs#153

TL;DR: It's not possible right now to support hotfixes for older version if there is a major version after it.

This would be possible if we handle tag sorting ourselves, but then tools like git describe won't work, and we're currently using this in OBS to easily name versions. The idea was to slowly expand on what we already had, in time we might introduce more improvements to the branching. Anyway, every feature we support in that regard would complicate the CI, log generation and mental burden, so it's good to have good reason for introducing it.

[gagandeepb]

Thanks @skrech. Looks good for now, let's talk about evolving the branching strategy as and when a need of sort I mentioned in my previous comment arises (i.e. if we really need to release a hotfix for an old major version).