Cruise Control REST API cannot be reached using Kubernetes Ingress

[fzlrhmn]

Hi everyone,

I am setting up Kafka with cruise control enabled. Following is my kafka.yml with disabled ssl and security for Cruise Control.

apiVersion: kafka.strimzi.io/v1beta2
kind: Kafka
metadata:
  name: cluster-11
  namespace: kafka
spec:
  kafka:
    version: 3.6.0
    replicas: 3
    listeners:
      - name: plain
        port: 9092
        type: internal
        tls: false
      - name: tls
        port: 9093
        type: internal
        tls: true
      - name: external
        port: 9094
        type: ingress
        tls: true
        configuration:
          bootstrap:
            # bootstrap
          brokers:
              # brokers
    config: 
      # config ...
  zookeeper:
    replicas: 3
    storage:
      type: persistent-claim
      size: 10Gi
      deleteClaim: true
  entityOperator:
    topicOperator: {}
    userOperator: {}
  cruiseControl:
    config:
      webserver.security.enable: false
      webserver.ssl.enable: false

And also I apply for kafka rebalance using following yaml

apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaRebalance
metadata:
  name: cluster-11-rebalance
  namespace: kafka
  labels:
    strimzi.io/cluster: cluster-11
spec: {}

Once applied, I can see that all pods are running. and services is all up and running.

Now, the problem is, I cannot access Cruise Control REST API from other pods except cruise control pod. I am accessing using internal domain cluster-11-cruise-control:9090.

This one is result from cruise control pods, I got response immediately.

➜  ~ kubectl exec -it cluster-11-cruise-control-asdfasdfasdf-rdfed4 -- /bin/sh
sh-4.4$ curl cluster-11-cruise-control:9090/kafkacruisecontrol/kafka_cluster_state?json=true
{"KafkaPartitionState":{"offline":[],"urp":[],"with-offline-replicas":[],"under-min-isr":[]},"KafkaBrokerState":{"IsController":{"0":false,"1":true,"2":false},"Summary":{"StdLeadersPerBroker":0.0,"Leaders":180,"MaxLeadersPerBroker":60,"Topics":12,"MaxReplicasPerBroker":159,"StdReplicasPerBroker":0.4714045207910317,"Brokers":3,"AvgReplicationFactor":2.6444444444444444,"AvgLeadersPerBroker":60.0,"Replicas":476,"AvgReplicasPerBroker":158.66666666666666},"OutOfSyncCountByBrokerId":{},"LeaderCountByBrokerId":{"0":60,"1":60,"2":60},"OnlineLogDirsByBrokerId":{"0":["/var/lib/kafka/data-0/kafka-log0"],"1":["/var/lib/kafka/data-0/kafka-log1"],"2":["/var/lib/kafka/data-0/kafka-log2"]},"BrokerSetByBrokerId":{},"OfflineLogDirsByBrokerId":{"0":[],"1":[],"2":[]},"ReplicaCountByBrokerId":{"0":159,"1":158,"2":159},"OfflineReplicaCountByBrokerId":{}},"version":1}sh-4.4$

But if I access it from other pod, let say cluster-11-entity-operator-575d7c5ff8-6v4pv, it will have connection timed out.

➜  ~ kubectl exec -it cluster-11-entity-operator-575d7c5ff8-6v4pv -- /bin/sh
Defaulted container "topic-operator" out of: topic-operator, user-operator, tls-sidecar
sh-4.4$ curl cluster-11-cruise-control:9090/kafkacruisecontrol/kafka_cluster_state?json=true
curl: (7) Failed to connect to cluster-11-cruise-control port 9090: Connection timed out

With this issue, I cannot expose Cruise Control Rest API with Ingress. Any ingress that point to cruise control service, will have connection timed out. Any suggestion?

[scholzj]

In general, we do not provide access to the REST API. Using the REST API might conflict with the operator and cause issues. The right way to use Cruise Control is using the KafkaRebalance resource.

That said, one of the things you can check are network policies -> did you create some network policy to allow you access to the REST API?

[fzlrhmn]

Thanks for the answer. What kind of conflict that may happened? I am planning to use the REST API only for GET request mostly for monitoring purposes and getting kafka cluster state.

I have been looking into the network policies, haven't done it yet. But seems I need to remove the cruise control policy that only allow incoming request from cluster-operator. Is that right?

[scholzj]

GET requests should be fine. But if you would use Cruise Control independently, it might for example interfere with rolling updates or scaledowns etc.

As for the network policy -> you cannot delete the one Strimzi creates. That is there to ensure Strimzi can communicate with Cruise Control. But you can add your own policy that adds your own rules for your application to access it.

[fzlrhmn]

I don't have any plan to use Cruise Control REST API as KafkaRebalance is easier to use. So I will keep it that way.

But you can add your own policy that adds your own rules for your application to access it.
I have tried this and seems this is the only way now. Thanks in advance.