Upgrade to rabbit 4.1.X

[YuryHrytsuk]

Following the incident https://git.speag.com/oSparc/osparc-infra/-/issues/70 we need to update to rabbit 4.1.x to avoid the same problem in the future

Note: as we use AWS Rabbit, we may want to consider a proper upgrade as simple deleta-all-start-from-scratch procedure is not applicable

Todo:

• use rabbit 4.1 in simcore stack (⬆️ upgrading rabbitmq to 4.1.2 ⚠️🚨 osparc-simcore#8109)
• upgrade aws rabbit --> 3.17 is latest supported (available) verison --> cannot be done

[YuryHrytsuk]

FYI @sanderegg

[YuryHrytsuk]

Version in use: 3.13.7 (both on-premise and AWS)

[YuryHrytsuk]

Version updates

History / Sequence:

• 3.13.7 --> 4.0.x --> 4.1.x (latest is 4.1.2)

We can upgrade directly from 3.13.7 to 4.1.2 but we still follow upgrade instructions from version 4.0.1 to be on safe side.

4.0.x release:

• https://github.com/rabbitmq/rabbitmq-server/releases/tag/v4.0.1

4.1.x release:

• https://github.com/rabbitmq/rabbitmq-server/releases/tag/v4.1.0

Upgrading guide

• https://www.rabbitmq.com/docs/upgrade

Deprecated Features

Overview / Documentation https://www.rabbitmq.com/docs/deprecated-features
List of deprecated features https://www.rabbitmq.com/release-information/deprecated-features-list

[YuryHrytsuk]

Breaking changes (3.13.7 --> 4.0.1)

Get more information at https://github.com/rabbitmq/rabbitmq-server/releases/tag/v4.0.1

To check if we are not affected:

• Classic Queues is Now a Non-Replicated Queue Type
  • If nodes is down, we lose messages (from Classical Queues) @YuryHrytsuk double check if it holds --> it does
  • tracked in RabbitMQ: properly configure queues for HA osparc-simcore#8078
• Quorum Queues Now Have a Default Redelivery Limit
• AMQP 0.9.1 x-death header Rabbitmq dead letter: support both 3.13 and 4.1 rabbitmq versions osparc-simcore#8077
• CQv1 Storage Implementation was Removed
  • switch to classic queues version 2 happens automatically @YuryHrytsuk
  • https://www.rabbitmq.com/docs/4.0/classic-queues#migration
• Settings cluster_formation.randomized_startup_delay_range.* were removed --> we don't use it
• Default Maximum Message Size Reduced to 16 MiB
  • we shall be fine. If not set max_message_size = 33554432 in rabbitmt.conf
  • make sure our message are within new size boundaries @YuryHrytsuk @mguidon --> shall be withing boundaries
• AMQP 1.0 --> we use asyncio-pika (0.9 protocol)
• MQTT
  • Make sure user configuration is still valid
• TLS Client (LDAP, Shovels, Federation) Defaults
  • Probably this does not affect us. @YuryHrytsuk check
• Shovels
  • we don't use this plugin

To verify / do after update

• x-death header logic works properly @sanderegg @GitHK
• images / graphs can be sent as Logs via rabbit (lower than 16MB @mguidon @bisgaard-itis @YuryHrytsuk
• Several Disk I/O-Related Metrics were Removed --> update grafana dashboards if necessary Self-hosted RabbitMQ: update grafana dashboards #1151

[YuryHrytsuk]

Breaking changes (4.0.1 --> 4.1.0)

Get more information at https://github.com/rabbitmq/rabbitmq-server/releases/tag/v4.1.0

To check if we are not affected:

• Initial AMQP 0-9-1 Maximum Frame Size
  • not related to us
• Node.js amqplib Must Be Upgraded
  • we don't use this
• MQTT
  • same as in 4.0 upgrade
• etcd Peer Discovery
  • we do not use
• OAuth 2 AuthN and AuthZ Plugin --> we don't use
• Erlang AMQP 1.0 Client --> we don't use

[YuryHrytsuk]

Rabbit state

internal master

• All features are enabled ✅
• No deprecated features in use ✅

aws master (*)

• All features are enabled ✅
• Deprecated features are in use ⚠

---

dalco stag

• Not all features are enabled ⚠
• No deprecated features in use ✅

nih stag

• Not all features are enabled ⚠
• No deprecated features in use ✅

---

sim4life (*)

• All features are enabled ✅
• Deprecated features are in use ⚠

nih prod

• All features are enabled ✅
• No deprecated features in use ✅

aws tip

• Not all features are enabled ⚠
• No deprecated features in use ✅

dalco prod

• Not all features are enabled ⚠
• No deprecated features in use ✅

internal tip (prod)

• Not all features are enabled ⚠
• No deprecated features in use ✅

[YuryHrytsuk]

If we want to use streams with RabbitMQ we may need to take into account which protocols clients we use support

source: rabbitmq/rabbitmq-website#2165 (comment)

FYI @sanderegg