Additional NodePort change every second

[asoleimanibo]

Hello,
I'm using the RabbiMQ operator to running RabbitMQ on the Kubernetes cluster.
This is my definition.yaml:

apiVersion: rabbitmq.com/v1beta1
kind: RabbitmqCluster
metadata:
  name: rabbitmqcluster
spec:
  image: rabbitmq:3.8.9-management
  replicas: 3
  override:
    service:
      spec:
        type: NodePort
        ports:
          - name: additional-port # adds an additional port on the service
            protocol: TCP
            port: 32011
            targetPort: 32011
    statefulSet:
      spec:
        template:
          spec:
            containers:
              - name: rabbitmq
                ports:
                  - containerPort: 32011 # opens an additional port on the rabbitmq server container
                    name: additional-port
                    protocol: TCP
  imagePullSecrets:
  - name: rabbitmq-cluster-registry-access
  service:
    type: NodePort
  persistence:
    storageClassName: rbd
    storage: 20Gi
  resources:
    requests:
      cpu: 2000m
      memory: 6Gi
    limits:
      cpu: 2000m
      memory: 6Gi
  rabbitmq:
    additionalConfig: |
      log.console = true
      log.console.level = debug
      log.file = rabbit.log
      log.dir = /var/lib/rabbitmq/
      log.file.level = debug
    additionalPlugins:
      - rabbitmq_top
      - rabbitmq_shovel
      - rabbitmq_management
      - rabbitmq_peer_discovery_k8s
      - rabbitmq_stomp
      - rabbitmq_prometheus
      - rabbitmq_peer_discovery_consul
    envConfig: |
      RABBITMQ_LOGS="rabbit.log"

after applying this definition, the additional-port that was exposed from the cluster changed every second:

[mkuratczyk]

I can confirm that it happens for me as well. Operator's logs show that it keeps reconciling the services which explains why a new port is getting assigned over and over. While I agree that this shouldn't happen, can you please explain your use case to help us prioritize this issue? Why do need the additional port?

[asoleimanibo]

@mkuratczyk I want to set static ports for STOMP and AMQP NodePort.

[mkuratczyk]

You don't need any overrides for that. This is sufficient and works correctly for me:

apiVersion: rabbitmq.com/v1beta1
kind: RabbitmqCluster
metadata:
  name: nodeport
spec:
  replicas: 3
  rabbitmq:
    additionalPlugins:
      - rabbitmq_stomp
  service:
    type: NodePort

[asoleimanibo]

@mkuratczyk yes but I want to specify the port number because it will be changed after redeploying the cluster.

[mkuratczyk]

Ok, until we fix this issue, I can suggest two workarounds:

1. Services of type NodePort are visible as <NodeIP>:spec.ports[*].nodePort (what you are trying to do) but also as .spec.clusterIP:spec.ports[*].port (this is the standard port in this case - 5672, etc) - I guess you need to actually use the former but I thought I'd mention this
2. You can deploy an additional service that is not managed by the Operator. If you deploy such instances regularly, you can have a template that deploys a RabbitmqCluster resource as well as a Service to match it. I know it's not perfect but perhaps this will unblock you for now.

Also, for our better understanding - can you explain why you need to rely on NodePorts? I guess you hit this issue because NodePorts are rarely used (statically assigning ports to services doesn't sounds like fun), so it'd help us if we knew when people rely on them. Thank you,

[asoleimanibo]

@mkuratczyk Thank you so much for your help. We use NodePort for connecting our applications to the cluster located in the Kubernetes cluster for debugging, so we want a static port to prevent change the code.

[github-actions]

This issue has been marked as stale due to 60 days of inactivity. Stale issues will be closed after a further 30 days of inactivity; please remove the stale label in order to prevent this occurring.

[motmot80]

In our case we are using a sandbox dev deployment with minikube and rabbitmq. Running on windows using miniport and NodePort you can easily setup the dev environment against minikube.mshome.net:FIXED_PORT. This was possible for about ten other helm chart deployments (including activemq, ibm-mq, prometheus-op, cass-op, kafka-strimzi-op, ...) but not for the rabbitmq-cluster-operator.

A similar use case as with @soleimaniamir

[motmot80]

In our case locking the NodePort was possible:

[...]
  service:
    type: NodePort
  override:
    service:
      spec:
        ports:
        - name: tcp-amqp
          protocol: TCP
          port: 5672
          targetPort: 5672
          nodePort: 35672
[...]

Deployment result:

[winterrobert]

Had a use case where we were trying to run the rabbitmq operator in locally in k3d - initially tried to get it working with nodeports like this to get a consistent port for our devs but ended up using type: LoadBalancer instead.