TODO

• document how to put node under maintenance --> readme
• support single node cluster (for local or tiny deployments) --> done via jinja and iterating over node count
• document how to update erlang cookie (auth secret to access rabbit nodes with CLI client)
• document autoscaling (joining nodes dynamically on demand) --> not supported at the moment
• how to properly add / remove nodes? --> readme
• test rabbit node count >= 3 --> test repo config values unit test
• how to apply new settings in rabbitmq.conf on a running cluster --> not supported (more in readme)
  • avoid causing restart of containers because of config sha change --> drop sha part so that docker fails to update service on config change
• add e2e test monitoring health of the cluster --> ops e2e test added
• run haproxy highly available --> 2+ replicas running
• make down (reasonable behaviour) --> simply remove the stack but not volumes. Add extra target to clean volumes
• applying changes via CI Pipelines --> deploy rabibt job is added
  • start cluster fresh new with empty volumes add later if there is a need. otherwise rely on manual operations if it comes to it (manually using makefile targets)
• restarting (rabbitmq node) service --> document behaviour --> we have stacks. Nothing specific to document now

Clients should properly use HA rabbit

• configure default replication factor for quorum queues? --> via rabbitmq.conf
• how to connect to a multi-node cluster --> it is hidden by haproxy (loadbalancer) --> no changes
• for backenders: make sure clients retry connection on failure

Cluster Formation

Source https://www.rabbitmq.com/docs/clustering

Ways of Forming a Cluster

• Declaratively by listing cluster nodes in config file <--- we use
• Declaratively using DNS-based discovery
• Declaratively using AWS (EC2) instance discovery
• Declaratively using Kubernetes discovery
• Declaratively using Consul-based discovery
• Declaratively using etcd-based discovery

Node Names (Identifiers)

• must be unique --> achieved via docker service name and env variable

Cluster Formation Requirements

• every cluster member must be able to resolve hostnames of every other cluster member, its own hostname, as well as machines on which command line tools such as rabbitmqctl might be used --> docker swarm networking

Ports That Must Be Opened for Clustering and Replication --> all works by default in docker swarm (all ports allowed)

• 4369: epmd, a helper discovery daemon used by RabbitMQ nodes and CLI tools
• 6000 through 6500: used by RabbitMQ Stream replication
• 25672: used for inter-node and CLI tools communication and is allocated from a dynamic range (limited to a single port by default, computed as AMQP port + 20000)
• 35672-35682: used by CLI tools for communication with nodes and is allocated from a dynamic

Nodes in a Cluster

• Nodes are Equal Peers

For two nodes to be able to communicate they must have the same shared secret called the Erlang cookie.

• Erlang cookie generation should be done at cluster deployment stage ⚠️ --> achieved via common secret

Node Counts and Quorum:

• Two node clusters are highly recommended against --> added a test to forbit 2 cluster node configuraiton

Clustering and Clients

Messaging Protocols

• In case of a node failure, clients should be able to reconnect to a different node, recover their topology and continue operation --> Task for backenders
• Most client libraries accept a list of endpoints --> we use loadbalancer and 1 endpoint

Stream Clients

• RabbitMQ Stream protocol clients behave differently from messaging protocols clients --> unrelated for us

Queue and Stream Leader Replica Placement

• Queue leaders should be reasonably evenly distributed across cluster nodes (see this doc) --> we have rabbit connections evenly distributed + added comment to do changes if this (haproxy load balancing mechanism) is changed
  • https://www.rabbitmq.com/docs/quorum-queues#member-management

Cleaning volumes

• Avoid tasks taking unlimited space --> do no retry jobs + always remove stack before starting new tasks
• Avoid unexpected volume removal
  • Deleting volumes failed but tasks keep running --> do not retry jobs + use timeouts
  • Deleting volumes unrelated to rabbit (safeguards) --> added

HA Proxy highly available

• running 2+ replicas and statistics --> we do not expose / use statistics at the beginning