Questions about the release process

[sergiodj]

Hello,

I'm currently investigating the possibility of having Ubuntu offer micro (patch) release upgrades of rabbitmq-server to its users that are currently in Ubuntu LTS releases. I have a few questions about your release process, though. I hope it's OK to list them here, but please let me know if I should send them somewhere else.

1. According to https://www.rabbitmq.com/versions.html , it seems that the period which a release series stays in service for is not constant among releases. Some were in service for 18 months, others for 20+, and there are those that were in service for 34 months. It doesn't seem like there is a pattern to be followed here, but I may be wrong. Is this a correct assessment?

2. There seems to be quite a number of micro releases in each series. So far, all changes listed in the changelogs point to bug fixes or improvements to existing features. I would like to confirm that no new features are added between patch releases for the same series, and that no breaking changes (e.g., the deprecation of a configuration directive) are expected to be present in such cases either.

3. Aside from Github actions, is there any other CI/CD system in place to test new releases before they're made available?

4. During the Extended Commercial Support period of a series, I understand that only high severity issues reported by users with a commercial license will be acted upon. Are the patch releases created during this cycle made publicly available, or are they created just for the customer?

I believe that's all I have for now. Thank you in advance!

[michaelklishin]

1. It is very likely to be 18 months now, previously was about two years, and in the 3.6-3.7 era, more than that
2. It depends on what you consider a feature. Sometimes we add diagnostics commands or configuration features. Sometimes a bug fix requires a breaking change, although in modern versions such change will be put behind a feature flag in patch releases. Breaking changes in client protocols are extremely rare, I recall two in the last twelve years or so that required or could require a client library upgrade
3. Concourse, which is internal to the core team at VMware. It tests package installation, basic upgrades, and client library suites, then publishes alpha releases
4. Patch releases of OSS RabbitMQ during the extended commercial support period are regular patch releases, there is no alternative repository for paying customers

[sergiodj]

Thank you very much for the prompt reply, @michaelklishin .

The answer to the second point is a bit concerning to me because Ubuntu has a very strict policy forbidding breaking changes and/or new features to be released in stable Ubuntu releases. However, the "feature flag" approach seems sensible to me. That might be enough to solve the concerns that may be raised by part of the community.

I'm still investigating things and will post more questions here if needed. Thanks for the help.

[michaelklishin]

Well, you will need to find a way to enable the feature flags after upgrades, or else some fixes won't have any effect.

It's not a solved problem for installation mechanisms that are not aware of cluster-wide state, such as apt.

Our docs pretty explicitly suggest bypassing standard Debian and Ubuntu repositories for RabbitMQ and Erlang, as those distributions are always at least one, sometimes two or even three minors behind.
It would be nice to get least patch bumps for LTS releases but we are also OK with having to maintain and distribute our own packages.

[sergiodj]

If I understood the concept of "feature flags" correctly, then we will not want to enable them after upgrades at all, because that would mean enabling a new feature in a patch release, which is a no-no for Ubuntu stable release upgrades. But that shouldn't be much of a problem because it seems to me that the vast majority of fixes that are present in a patch release don't actually require "feature flags" to be enabled (exactly because these fixes are not new features). Am I understanding this correctly?

As for your documentation recommending users to not use the rabbitmq-package from Debian/Ubuntu, that's indeed (unfortunately) the case for many other upstream projects, and one of the main reasons why I'm working on formalizing the process to at least ship the patch releases to our users.

Thanks!

[mkuratczyk]

Indeed most fixes don't require feature flags. We've only had like a dozen feature flags in total, since introducing the feature flags mechanism in 2017. A vast majority of them was introduced in the .0 release of a new minor. I believe there were exactly 2 feature flags introduced to solve a bug. Unfortunately this may happen, since certain types of changes require all cluster members to follow either the old behaviour or the new (fixed) behaviour. But the worst case scenario is that users would get all the other fixes introduced in the patch release, just not the one gated by a feature flag (until they enable it).

If you don't want to automatically enable the feature flags, we'd appreciate if you could add a post-upgrade command that checks if there are any disabled feature flags (rabbitmqctl list_feature_flags | grep disabled$) an if yes - print a recommendation to the user to enable them.

[sergiodj]

Thank you very much, and yes, I was thinking along the lines of having some kind of message to users letting them know that there are disabled feature flags.

This gives me more confidence that it should indeed be possible to have patch releases shipped to Ubuntu users who are on LTS.