Deploy single Rabbit MQ cluster spread across multiple K8S clusters

[smuthreja]

Community Support Policy

• I have read RabbitMQ's Community Support Policy
• I run RabbitMQ 4.x, the only series currently covered by community support
• I promise to provide all relevant information (versions, logs from all nodes, rabbitmq-diagnostics output, detailed reproduction steps)

RabbitMQ version used

4.0.5

Erlang version used

27.2.x

Operating system (distribution) used

Ubuntu 22

How is RabbitMQ deployed?

Bitnami Helm chart

rabbitmq-diagnostics status output

See https://www.rabbitmq.com/docs/cli to learn how to use rabbitmq-diagnostics

2025-01-29 11:25:13.796756+00:00 [info] <0.3298.0> DB: checking if `rabbit@rabbitmq-c3-0.rabbitmq-c3-headless.rabbitmq.svc.cluster.local` can join cluster using remote node `rabbit@rabbitmq-c1-0.rabbitmq-c1-headless.rabbitmq.svc.cluster.local`
2025-01-29 11:25:13.796902+00:00 [notice] <0.3298.0> Feature flags: checking nodes `rabbit@rabbitmq-c3-0.rabbitmq-c3-headless.rabbitmq.svc.cluster.local` and `rabbit@rabbitmq-c1-0.rabbitmq-c1-headless.rabbitmq.svc.cluster.local` compatibility...
2025-01-29 11:25:13.819078+00:00 [error] <0.3298.0> Feature flags: error while running:
2025-01-29 11:25:13.819078+00:00 [error] <0.3298.0> Feature flags:   rabbit_ff_controller:running_nodes[]
2025-01-29 11:25:13.819078+00:00 [error] <0.3298.0> Feature flags: on node `rabbit@rabbitmq-c1-0.rabbitmq-c1-headless.rabbitmq.svc.cluster.local`:
2025-01-29 11:25:13.819078+00:00 [error] <0.3298.0> Feature flags:   exception error: {erpc,noconnection}
2025-01-29 11:25:13.819078+00:00 [error] <0.3298.0> Feature flags:     in function  erpc:call/5 (erpc.erl, line 710)
2025-01-29 11:25:13.819078+00:00 [error] <0.3298.0> Feature flags:     in call from rabbit_ff_controller:rpc_call/5 (rabbit_ff_controller.erl, line 1394)
2025-01-29 11:25:13.819078+00:00 [error] <0.3298.0> Feature flags:     in call from rabbit_ff_controller:list_nodes_clustered_with/1 (rabbit_ff_controller.erl, line 486)
2025-01-29 11:25:13.819078+00:00 [error] <0.3298.0> Feature flags:     in call from rabbit_ff_controller:check_node_compatibility_task/2 (rabbit_ff_controller.erl, line 398)
2025-01-29 11:25:13.819078+00:00 [error] <0.3298.0> Feature flags:     in call from rabbit_db_cluster:can_join/1 (rabbit_db_cluster.erl, line 60)
2025-01-29 11:25:13.819078+00:00 [error] <0.3298.0> Feature flags:     in call from rabbit_db_cluster:join/2 (rabbit_db_cluster.erl, line 92)
2025-01-29 11:25:13.819078+00:00 [error] <0.3298.0> Feature flags:     in call from erpc:execute_call/4 (erpc.erl, line 589)
2025-01-29 11:25:13.819078+00:00 [error] <0.3298.0> 
2025-01-29 11:25:13.819460+00:00 [warning] <0.3298.0> Feature flags: error while querying cluster members from node `rabbit@rabbitmq-c1-0.rabbitmq-c1-headless.rabbitmq.svc.cluster.local`:
2025-01-29 11:25:13.819460+00:00 [warning] <0.3298.0> {error,{erpc,noconnection}}

Logs from node 1 (with sensitive values edited out)

See https://www.rabbitmq.com/docs/logging to learn how to collect logs

2025-01-29 11:25:13.796756+00:00 [info] <0.3298.0> DB: checking if `rabbit@rabbitmq-c3-0.rabbitmq-c3-headless.rabbitmq.svc.cluster.local` can join cluster using remote node `rabbit@rabbitmq-c1-0.rabbitmq-c1-headless.rabbitmq.svc.cluster.local`
2025-01-29 11:25:13.796902+00:00 [notice] <0.3298.0> Feature flags: checking nodes `rabbit@rabbitmq-c3-0.rabbitmq-c3-headless.rabbitmq.svc.cluster.local` and `rabbit@rabbitmq-c1-0.rabbitmq-c1-headless.rabbitmq.svc.cluster.local` compatibility...
2025-01-29 11:25:13.819078+00:00 [error] <0.3298.0> Feature flags: error while running:
2025-01-29 11:25:13.819078+00:00 [error] <0.3298.0> Feature flags:   rabbit_ff_controller:running_nodes[]
2025-01-29 11:25:13.819078+00:00 [error] <0.3298.0> Feature flags: on node `rabbit@rabbitmq-c1-0.rabbitmq-c1-headless.rabbitmq.svc.cluster.local`:
2025-01-29 11:25:13.819078+00:00 [error] <0.3298.0> Feature flags:   exception error: {erpc,noconnection}
2025-01-29 11:25:13.819078+00:00 [error] <0.3298.0> Feature flags:     in function  erpc:call/5 (erpc.erl, line 710)
2025-01-29 11:25:13.819078+00:00 [error] <0.3298.0> Feature flags:     in call from rabbit_ff_controller:rpc_call/5 (rabbit_ff_controller.erl, line 1394)
2025-01-29 11:25:13.819078+00:00 [error] <0.3298.0> Feature flags:     in call from rabbit_ff_controller:list_nodes_clustered_with/1 (rabbit_ff_controller.erl, line 486)
2025-01-29 11:25:13.819078+00:00 [error] <0.3298.0> Feature flags:     in call from rabbit_ff_controller:check_node_compatibility_task/2 (rabbit_ff_controller.erl, line 398)
2025-01-29 11:25:13.819078+00:00 [error] <0.3298.0> Feature flags:     in call from rabbit_db_cluster:can_join/1 (rabbit_db_cluster.erl, line 60)
2025-01-29 11:25:13.819078+00:00 [error] <0.3298.0> Feature flags:     in call from rabbit_db_cluster:join/2 (rabbit_db_cluster.erl, line 92)
2025-01-29 11:25:13.819078+00:00 [error] <0.3298.0> Feature flags:     in call from erpc:execute_call/4 (erpc.erl, line 589)
2025-01-29 11:25:13.819078+00:00 [error] <0.3298.0> 
2025-01-29 11:25:13.819460+00:00 [warning] <0.3298.0> Feature flags: error while querying cluster members from node `rabbit@rabbitmq-c1-0.rabbitmq-c1-headless.rabbitmq.svc.cluster.local`:
2025-01-29 11:25:13.819460+00:00 [warning] <0.3298.0> {error,{erpc,noconnection}}

Logs from node 2 (if applicable, with sensitive values edited out)

See https://www.rabbitmq.com/docs/logging to learn how to collect logs

# PASTE LOG HERE, BETWEEN BACKTICKS

Logs from node 3 (if applicable, with sensitive values edited out)

See https://www.rabbitmq.com/docs/logging to learn how to collect logs

# PASTE LOG HERE, BETWEEN BACKTICKS

rabbitmq.conf

See https://www.rabbitmq.com/docs/configure#config-location to learn how to find rabbitmq.conf file location

## Username and password
default_user = user
## Clustering
##
cluster_name = rabbitmq-c3
cluster_formation.peer_discovery_backend  = rabbit_peer_discovery_k8s
cluster_formation.k8s.host = kubernetes.default
cluster_formation.k8s.address_type = hostname
cluster_formation.k8s.service_name = rabbitmq-c3-headless
cluster_formation.k8s.hostname_suffix = .rabbitmq-c3-headless.rabbitmq.svc.cluster.local
cluster_formation.node_cleanup.interval = 10
cluster_formation.node_cleanup.only_log_warning = true
cluster_partition_handling = autoheal

# queue master locator
queue_master_locator = min-masters
# enable loopback user
loopback_users.user = false
#default_vhost = rabbitmq-vhost
#disk_free_limit.absolute = 50MB
## Prometheus metrics
##
prometheus.tcp.port = 9419
## TCP Listen Options
##
tcp_listen_options.backlog = 128
tcp_listen_options.nodelay = true
tcp_listen_options.linger.on      = true
tcp_listen_options.linger.timeout = 0
tcp_listen_options.keepalive = false

Steps to deploy RabbitMQ cluster

Deploy RabbitMQ on 2 K8S clusters using helm:

helm install rabbitmq-c1 bitnami/rabbitmq --namespace rabbitmq --context="${CTX_CLUSTER1}"

helm install rabbitmq-c3 bitnami/rabbitmq --namespace rabbitmq --context="${CTX_CLUSTER2}"

I already have ISTIO Mesh installed to connect 2 clusters with different networks.

istio-injection=enabled has been enabled in rabbitmq namespace
I can also see that istioctl endpoint lists the SVCs from local & remote cluster.

Steps to reproduce the behavior in question

1. Install ISTIO Mesh with Multi Primary on different networks. Follow guide: https://istio.io/latest/docs/setup/install/multicluster/multi-primary_multi-network/
2. Create Namespace "rabbitmq" and enable istio-injection on both clusters.
3. Install RabbitMQ cluster on both K8S clusters with different name.
4. Run join cluster from Cluster-1 to Cluster-2 and vice-versa

advanced.config

See https://www.rabbitmq.com/docs/configure#config-location to learn how to find advanced.config file location

# PASTE advanced.config HERE, BETWEEN BACKTICKS

Application code

# PASTE CODE HERE, BETWEEN BACKTICKS

Kubernetes deployment file

# Relevant parts of K8S deployment that demonstrate how RabbitMQ is deployed
# PASTE YAML HERE, BETWEEN BACKTICKS

What problem are you trying to solve?

We need one RabbitMQ cluster spread across multiple k8s clusters.
Our requirement is that only one consumer consume the messages from this RabbitMQ cluster.

Using Federation/Shovel, its more of duplicate queues are being created across clusters where there is no guarantee that only one consumer will consume the message. So, wanted to avoid this Federation/Shovel Plugin.

[mkuratczyk]

You didn't say why you "need" that, but most likely - you don't. To be honest, it just sounds like a bad idea.

If the goal is to achieve a disaster recovery solution - we have a commercial feature, Warm Standby Replication for such use cases, but it still replicates data between two different RabbitMQ clusters (most likely each deployed to a different Kubernetes cluster, in a different AZ).

If you want to explore spreading a RabbitMQ cluster across Kubernetes clusters - go ahead, but we won't spend time on this.

[vovtz]

We have the same requirement. @mkuratczyk, not sure about @smuthreja’s requirements, but we are definitely not trying to accomplish disaster recovery. Federation would cover that requirement, but -as @smuthreja pointed out- if consumers are connected to both (in fact mostly independent) RabbitMQ clusters, they will consume the same messages.

What we’re after is a hot-hot failover between two Rabbit clusters or -even better- two Rabbit clusters that cooperatively share all their nodes.

[mkuratczyk]

I'm afraid I don't understand what you mean based on this brief description. If you would like to discuss the idea, please provide a (much) more detailed description, including the goals, alternatives you considered and some examples of what should happen in what situation.

[michaelklishin]

And that more detailed description belongs into a separate discussion.