RabbitMQ Upgrade 3.8.9 to 3.10.8

[alcibiades34]

Hello everyone, I have a 3 nodes RabbitMQ cluster that running as containers on the servers. The current version of my cluster is 3.8.9 and I want to upgrade it to 3.10.8. But I could not see the RabbitMQ version Upgradability about 3.8 to 3.10 on the documentation.

The question is that can I upgrade the rabbitmq 3.8.9 to 3.10.8 directly, without upgrading 3.8.9 to 3.9 first? Is this possible and will it cause a problem?

Thank you.

[inikulshin]

I tried to upgrade directly from 3.8.27 (in tests) and it didn't succeed at 100% (some mnesia corruption, afair). So I added an upgrade to 3.9.22 as an intermediate.

But I have related questions about this intermediate version:

1. It's pretty obvious, that RabbitMQ service should start to complete the upgrade. But when it is safe to stop and uninstall it? Is rabbitmq-diagnostics.bat is_running enough?
2. Are there any requirements for intermediate rabbitmq.config file? It can be incompatible with the one from 3.8.x (e.g. logger configuration) and potentially with 3.10.x. Is []. good enough to upgrade process?
3. Are there any requirements for enabled plugins list? Is it safe to disable all of them with rabbitmq-plugins.bat set --offline before uninstalling 3.8.x and enable them back after installing 3.10.x?

[michaelklishin]

1. Unless you have a lot of on-disk data to recover, basically, yes. Some upgrades will require you to enable all feature flags (e.g. to 3.11.x), so that might have to happen before you shut a node down for the next upgrade.
2. I cannot think of 3.8 settings in the Erlang term config format that would break in 3.10. Same for rabbitmq.com. An empty config or no config file at all would do no harm.
3. Unless you have 3rd party plugins that are incompatible with the newer version, no need to disable anything. Incompatible plugins would have to be upgraded or disabled + removed from the plugins directory, then re-enabled at the end of the upgrade.

[inikulshin]

1. I think, there is should be some deterministic check, maybe some ugly rabbitmqctl eval? "a lot of on-disk data to recover" is not something that can be used in automatic upgrade script.
2. Logger configuration in 3.8.x:

{log, [
	{file, [
		{file, "[email protected]"},
		{formatter_config, [date," ",time," [",severity,"] ",{pid,[]}," ",message,"\r\n"]}
	]}
]}

vs 3.9.x and 3.10.x (to preserve old formatting as possible):

{log, [
	{file, [
		{file, "[email protected]"},
		{formatter, {
			rabbit_logger_text_fmt,
			#{
				time_format => {local, "~4..0b-~2..0b-~2..0b ~2..0b:~2..0b:~2..0b.~3..0b", [year, month, day, hour, minute, second, {second_fractional, 3}]},
				level_format => lc,
				single_line => false,
				line_format => [msg,"\r"],
				prefix_format => [time," [",level,"] ",pid," "]}
		}}
	]}
]}

It's very different and 3.9.x fails to start with the this configuration from 3.8.x.

3. We do have custom plugins, but fortunately they do not require any upgrade, only to be recompiled with matching Erlang version. Once there was a problem in upgrade: during initial service start inside the Windows installer RABBITMQ_PLUGINS directory couldn't contain custom plugins. But since windows-exe: optional sections in silent (/S) mode, bug fixes rabbitmq-packaging#15 the service start can be disabled.

---

disabled + removed

Right, disabling only is not enough. Thanks!

[lukebakken]

rabbitmq/rabbitmq-website#1533