RabbitMQ Cluster fails with error no endpoints available

[sathishc58]

I am trying to setup RabbitMQ Operator + RabbitMQ Cluster on K8S using Terraform

I installed cert-manager and rabbitmq-cluster-operator successfully. I am trying to create RabbitMQCluster using the following link

However RabbitMQCluster failed with the following message and output is given below

Message:               The service has no endpoints available
Reason:                NoEndpointsAvailable

Output (Tailored output)

[root@re-ctrl02 rbmq]# kubectl describe RabbitmqCluster -n rabbitmq-system
Name:         basic-cluster
Namespace:    rabbitmq-system
Labels:       <none>
Annotations:  <none>
API Version:  rabbitmq.com/v1beta1
Kind:         RabbitmqCluster
Spec:
  Image:  rabbitmq:3.8.18-management
  Override:
    Stateful Set:
      Spec:
        Template:
          Spec:
            Containers:
              Name:  rabbitmq
              Resources:
              Name:  rabbitmq
              Resources:
  Rabbitmq:
    Additional Config:                    hostname = "https://localhost"
                  host = "https://localhost"
                  port = "5672"
                  nodes = "rabbit@localhost"
                  default_vhost = "/"
                  default_user = "admin"
                  default_pass = "admin"
                  default_user_tags.administrator = true

  Replicas:  1
  Service:
    Type:                            NodePort
  Termination Grace Period Seconds:  604800
  Tls:
Status:
  Binding:
    Name:  basic-cluster-default-user
  Conditions:
    Last Transition Time:  2021-08-09T13:51:42Z
    Message:               0/1 Pods ready
    Reason:                NotAllPodsReady
    Status:                False
    Type:                  AllReplicasReady
    Last Transition Time:  2021-08-09T13:51:42Z
    Message:               The service has no endpoints available
    Reason:                NoEndpointsAvailable
    Status:                False
    Type:                  ClusterAvailable
    Last Transition Time:  2021-08-09T13:51:42Z
    Reason:                NoWarnings
    Status:                True
    Type:                  NoWarnings
    Last Transition Time:  2021-08-09T13:51:42Z
    Reason:                Initialising
    Status:                Unknown
    Type:                  ReconcileSuccess

I hope it complains about the following rabbitmq-services and they don't have endpoints. However, I am not sure what endpoint is expected and how to add it to services

[root@re-ctrl02 rbmq]# kubectl get svc -n rabbitmq-system                   NAME                  TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)                                          AGE
basic-cluster         NodePort    10.104.166.10   <none>        5672:32752/TCP,15672:32697/TCP,15692:32420/TCP   50s
basic-cluster-nodes   ClusterIP   None            <none>        4369/TCP,25672/TCP                               50s


[root@re-ctrl02 rbmq]# kubectl get ep -n rabbitmq-system
NAME                  ENDPOINTS   AGE
basic-cluster         <none>      58s
basic-cluster-nodes   <none>      58s

Kindly let me know what am I doing wrong

[mkuratczyk]

Remove your spec.rabbitmq.additionalConfig section of the definition - these are all unnecessary and most likely cause the problem (because you are setting localhost where it should be a real hostname). The Operator handles all the basic configuration automatically, including networking, clustering and creation of the default user (https://www.rabbitmq.com/kubernetes/operator/using-operator.html#creds). Similarily, I'm not sure what you have in the override but you don't need to override anything unless you have a very specific reason, so skip that part to get started.

[sathishc58]

@mkuratczyk Thanks for replying. Initially I did not set anything. After looking at the error message I tried adding those entries host and hostname. Despite removing the entries the error still persists

At the same time, I would like to understand about endpoints. How the endpoints are populated and where endpoints should point to ?

[mkuratczyk]

Endpoints will be automatically populated once RabbitMQ starts correctly. Don't worry about them - you should not need to configure anything around Endpoints. Please see the other comments for the next step

[sathishc58]

@mkuratczyk @mkuratczyk Current override section is mentioned below. I added section 'initContainer' to avoid epmd host error and permission denied issue on /var/data/mnesia

override = {
               "statefulSet" = {
                   "spec" = {
                        "template" = {
                            "metadata" = {
                                labels = var.labels
                            }

                          "spec" = {
                              containers = [ {
                                    name = "rabbitmq"
                                    image = var.image
                                 } ]
                              initContainers = [ {
                                    name = "rabbitmq"
                                    command = [ "sh", "-c", "cp /tmp/rabbitmq/rabbitmq.conf /etc/rabbitmq/rabbitmq.conf && chown 999:999 /etc/rabbitmq/rabbitmq.conf && echo '' >> /etc/rabbitmq/rabbitmq.conf ; cp /tmp/rabbitmq/advanced.config /etc/rabbitmq/advanced.config && chown 999:999 /etc/rabbitmq/advanced.config ; cp /tmp/rabbitmq/rabbitmq-env.conf /etc/rabbitmq/rabbitmq-env.conf && chown 999:999 /etc/rabbitmq/rabbitmq-env.conf ; cp /tmp/erlang-cookie-secret/.erlang.cookie /var/lib/rabbitmq/.erlang.cookie && chown 999:999 /var/lib/rabbitmq/.erlang.cookie && chmod 600 /var/lib/rabbitmq/.erlang.cookie ; cp /tmp/rabbitmq-plugins/enabled_plugins /etc/rabbitmq/enabled_plugins && chown 999:999 /etc/rabbitmq/enabled_plugins ; chown 999:999 /var/lib/rabbitmq/mnesia/" ]
                              } ]
                              hostAliases = [
                                  {
                                      ip = "127.0.0.1"
                                      hostnames = [ "basic-cluster-server-0.basic-cluster-nodes.rabbitmq-system" ]
                                  }
                              ]
                           }
                       }
                   }
               }
           }

[sathishc58]

@mkuratczyk Ok I'll try from my end. I'll keep you posted tomorrow.

[sathishc58]

@mkuratczyk This morning I started things afresh. I removed sections initContainer and additional_plugins and executed it

Note: I still had to add PVC, PV and hostAliases to overcome volume binding and epmd host issues

As mentioned by you I did not face permission denied issue but now I get the following error in pod logs

Pod logs

2021-08-11 13:53:44.601 [error] <0.273.0> Failed to fetch a list of nodes from Kubernetes API: {failed_connect,[{to_address,{"kubernetes.default",443}},
                 {inet,[inet],nxdomain}]}
2021-08-11 13:53:52.603 [error] <0.273.0> Failed to lock with peer discovery backend rabbit_peer_discovery_k8s: "{failed_connect,[{to_address,{\"kubernetes.default\",443}},\n                 {inet,[inet],nxdomain}]}"

[mkuratczyk]

1. Do you understand why epmd fails without your changes? It would be good to see if we can change something in the operator to make it work without user intervention (note: it does work without user intervention for almost everyone else so this problem seems specific to your environment)
2. The operator generates peer discovery configuration and as part of that, sets cluster_formation.k8s.host = kubernetes.default. Seems like that name should be different in your case. More info: https://kubernetes.io/docs/tasks/run-application/access-api-from-pod/#directly-accessing-the-rest-api

[sathishc58]

@mkuratczyk I was able to bring up RabbitMQ cluster successfully. However, it appears most of the people try on Cloud instead of bare-metal and things are bit easier on Cloud

[root@re-devk8s-control02 rbmq]# kubectl get pods -n rabbitmq-system
NAME                                         READY   STATUS    RESTARTS   AGE
basic-cluster-server-0                       1/1     Running   10         15h
rabbitmq-cluster-operator-85b59b4c6c-gwwdk   1/1     Running   1          2d15h

1. To my understanding RabbitMQ Cluster expects the following entry to be present in /etc/hosts file (during startup) inside pod's container and this is to eliminate 'epmd host error'

# Entries added by HostAliases.
127.0.0.1       basic-cluster-server-0.basic-cluster-nodes.rabbitmq-system

And kindly let me know how to create a queue dynamically. For example I was able to create users and assign them permissions using the following

            rabbitmq = {
                additionalConfig  = <<EOF
                  default_vhost = "/"
                  default_user = "admin"
                  default_pass = "admin"
                  default_user_tags.administrator = true
                  cluster_formation.k8s.host = kubernetes.default.svc.cluster.local
                EOF
            }

Similarly I would like to create queues, binding and exchanges using yaml or json instead of using rabbitmqctl utility

Last but not least, thanks a ton for your timely response

[mkuratczyk]

Having the hostname set up correctly is indeed necessary. I don't think we can change that.

For declaring exchanges, queues and bindings, you have multiple options:

1. Most commonly, they are defined by the applications themselves so no need to pre-declare them.
2. If you want to pre-declare them before connecting your applications, you can:

• use Topology Operator: https://github.com/rabbitmq/messaging-topology-operator/
• use import-definitions feature https://github.com/rabbitmq/cluster-operator/tree/main/docs/examples/import-definitions
• use the new feature of RabbitMQ 3.9.4 which added an option to import definitions from an HTTP endpoint https://github.com/rabbitmq/rabbitmq-server/releases/tag/v3.9.4