Assigning pre-allocated Private IP into the kafka configuration for azure internal LB

[vanyagyosheva]

Hello,
I'm working on implementing kafka with strimzi on azure aks clusters. We found that every time we apply terraform scripts , the LB ip is changing and we need to recreate the dns entry in azure. If we create strimzi kafka for the first time we don't have the private LB IP allocated , so we cannot specify it in advance in the kafka manifest. Could you suggest a way to pre allocate the IP before the kafka resource is applied:
this is my kafka.yaml
You can see that we have vars: loadBalancerIP: ${bootstrap_ip} and loadBalancerIP: ${broker_ip} which we can only use when it has been created once but if we run the installation from the beggining we don't have these IP. We don't want to create subnet again with terraform since that already exists and no need to change , could you suggest some way to create the private IP before applying kafka resource?

apiVersion: kafka.strimzi.io/v1beta2
kind: Kafka
metadata:
  name: kafka-svc
  namespace: ${namespace}
spec:
  kafka:
    version: 3.4.0
    replicas: 1
    resources:
      requests:
        memory: 4Gi
        cpu: "1"
      limits:
        memory: 4Gi
        cpu: "2"
    jvmOptions:
      -Xms: 3276m
      -Xmx: 3276m
    logging:
      type: inline
      loggers:
        logger.clients.level: INFO
        logger.toc.level: INFO
        logger.toc.name: io.strimzi.operator.topic.TopicOperator
        logger.top.level: INFO
        logger.top.name: io.strimzi.operator.topic
        rootLogger.level: INFO
        log4j.logger.kafka.controller: INFO
        log4j.logger.org.apache.kafka.controller: INFO
        log4j.logger.kafka.log.LogCleaner: INFO
        log4j.logger.state.change.logger: INFO
    listeners:
      - name: tls
        port: 9093
        type: internal
        tls: true
        authentication:
          type: tls
      - name: external
        port: 9092
        type: loadbalancer
        tls: false
        configuration:
          bootstrap:
            annotations:
              service.beta.kubernetes.io/azure-load-balancer-internal: "true"
              loadBalancerIP: ${bootstrap_ip}
          brokers:
            - broker: 0
              annotations:
                service.beta.kubernetes.io/azure-load-balancer-internal: "true"
                loadBalancerIP: ${broker_ip}
    template:
      externalBootstrapService:
        metadata:
          annotations:
            service.beta.kubernetes.io/azure-load-balancer-internal: "true"
      perPodService:
        metadata:
          annotations:
            service.beta.kubernetes.io/azure-load-balancer-internal: "true"
      pod:
        tmpDirSizeLimit: 5Gi
        affinity:
          podAntiAffinity:
            requiredDuringSchedulingIgnoredDuringExecution:
              - labelSelector:
                  matchExpressions:
                    - key: strimzi.io/name
                      operator: In
                      values:
                        - kafka-svc-kafka
                topologyKey: "kubernetes.io/hostname"
        tolerations:
          - key: "${toleration.key}"
            operator: Equal
            value: "yes"
            effect: "${toleration.effect}"
    livenessProbe:
      failureThreshold: 3
      initialDelaySeconds: 15
      periodSeconds: 10
      timeoutSeconds: 5
    readinessProbe:
      failureThreshold: 3
      initialDelaySeconds: 15
      periodSeconds: 10
      timeoutSeconds: 5
    config:
      offsets.topic.replication.factor: 1
      offsets.retention.minutes: 5259600
      transaction.state.log.replication.factor: 1
      transaction.state.log.min.isr: 1
      default.replication.factor: 1
      min.insync.replicas: 1
      inter.broker.protocol.version: "3.4"
      zookeeper.connection.timeout.ms: 120000
      log.retention.check.interval.ms: 300000
      log.retention.hours: ${log_retention_hours}
      log.segment.bytes: 1073741824
      log.cleaner.enable: true
      log.flush.interval.messages: 50000
      log.flush.interval.ms: 10000
      message.max.bytes: 100000000
      max.poll.interval.ms: 600000
      group.initial.rebalance.delay.ms: 0
      group.min.session.timeout.ms: 6000
      group.max.session.timeout.ms: 1800000
      heartbeat.interval.ms: 10000
      session.timeout.ms: 31000
      max.partition.fetch.bytes: 100000000
      fetch.max.bytes: 100000000
      replica.fetch.max.bytes: 3303304
      delivery.timeout.ms: 300000
      num.io.threads: 8
      num.network.threads: 5
      num.partitions: 8
      num.recovery.threads.per.data.dir: 2
      socket.receive.buffer.bytes: 31460040
      socket.request.max.bytes: 104857600
      socket.send.buffer.bytes: 31460040
      ssl.cipher.suites: TLS_AES_256_GCM_SHA384, TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
      ssl.enabled.protocols: TLSv1.2
      ssl.endpoint.identification.algorithm: HTTPS
      ssl.protocol: TLSv1.2
    storage:
      type: jbod
      volumes:
        - id: 0
          type: persistent-claim
          size: 20Gi
          class: managed-csi
          deleteClaim: true
        - id: 1
          type: persistent-claim
          size: 20Gi
          class: managed-csi
          deleteClaim: true
        - id: 2
          type: persistent-claim
          size: 20Gi
          class: managed-csi
          deleteClaim: true
        - id: 3
          type: persistent-claim
          size: 20Gi
          class: managed-csi
          deleteClaim: true
    metricsConfig:
      type: jmxPrometheusExporter
      valueFrom:
        configMapKeyRef:
          name: kafka-metrics
          key: kafka-metrics-config.yml
  zookeeper:
    replicas: 3
    resources:
      requests:
        memory: 2Gi
        cpu: "1"
      limits:
        memory: 2Gi
        cpu: "1.5"
    storage:
      size: 10Gi
      type: persistent-claim
      class: managed-csi
    logging:
      type: inline
      loggers:
        zookeeper.root.logger: "INFO"
    livenessProbe:
      failureThreshold: 3
      initialDelaySeconds: 15
      periodSeconds: 10
      timeoutSeconds: 5
    readinessProbe:
      failureThreshold: 3
      initialDelaySeconds: 15
      periodSeconds: 10
      timeoutSeconds: 5
    metricsConfig:
      type: jmxPrometheusExporter
      valueFrom:
        configMapKeyRef:
          name: kafka-metrics
          key: zookeeper-metrics-config.yml
    template:
      pod:
        affinity:
          podAntiAffinity:
            requiredDuringSchedulingIgnoredDuringExecution:
              - labelSelector:
                  matchExpressions:
                    - key: strimzi.io/name
                      operator: In
                      values:
                        - kafka-svc-zookeeper
                topologyKey: "kubernetes.io/hostname"
        tolerations:
          - key: "${toleration.key}"
            operator: Equal
            value: "yes"
            effect: "${toleration.effect}"
  entityOperator:
    template:
      pod:
        tolerations:
          - key: "${toleration.key}"
            operator: Equal
            value: "yes"
            effect: "${toleration.effect}"
    topicOperator:
      resources:
        requests:
          memory: 2Gi
          cpu: "1"
        limits:
          memory: 2Gi
          cpu: "1"
    userOperator: {}
  kafkaExporter:
    template:
      pod:
        tolerations:
          - key: "${toleration.key}"
            operator: Equal
            value: "yes"
            effect: "${toleration.effect}"

[scholzj]

The YAML is not-readable because it is not formatted as code. So as a first thing, you should probably format it properly so that it can be read and verified. Second, please keep in mind that all Strimzi can do is pass the IP into the service object. How Kubernetes provisions the actual loadbalancer is up to Kubernetes.

[vanyagyosheva]

Now the formatting should be fine. If we terrraform apply the LB before kafka and pass the IP to the configuration, could we also specify the load balancer name in kafka so it will not try to create new LB again?

[scholzj]

The CR looks good to me. Just as a sidenote, having four 20Gi volumes seems wrong. But that is not related to the load balancers of course.

What does it mean to terrraform apply the LB before kafka?

[vanyagyosheva]

Ah i accept other suggestions as well , so we can improve the strimzi kafka performance and state. Why 20Gi looks wrong?
regarding LB, i mean if we create the LB and get the IP we can pass them to the kafka config, but i'm not sure if it will not try to create new LB again , so we need to be sure that the kafka config will not create but use the existing one , so is there a way to specify the LB name in the kafka config?

[scholzj]

Why 20Gi looks wrong?

It's not that 20Gi on its own is wrong. But using many small disks does not make sense. It means that you need to worry about balancing the data between them and making sure no partition is bigger than 20Gi. With one 80Gi disk, this would be much easier. Using multiple disks is fine, but it makes more sense with much bigger sizes. E.g. when your disks are several Ti big, then more of them might make sense. You can also check this for more info: https://youtu.be/7gaOUjkuR68

regarding LB, i mean if we create the LB and get the IP we can pass them to the kafka config, but i'm not sure if it will not try to create new LB again , so we need to be sure that the kafka config will not create but use the existing one , so is there a way to specify the LB name in the kafka config?

The way this normally works is that:

• You configure the load balancer IP in the Kafka CR as you did
• Strimzi creates the Kubernetes Service which lists this IP address
• Kubernetes provisions the load balancer with this IP address (if supported in your environment)

Now, if you provision the Loadbalancers upfront with Terraform, how do you make sure Kubernetes will bind the right LB for the right service? I suspect it simply provisions a new LB and because the desired IP address is already taken it will use another one?