[Bug]: kafka use loadbalancer in kmm2 tls example not work #9864

usernameisnull · 2024-03-22T01:46:43Z

usernameisnull
Mar 22, 2024

Bug Description

use this example file: https://github.com/strimzi/strimzi-kafka-operator/blob/main/examples/security/tls-auth/mirror-maker-2.yaml changes:

the storage to local-path
spec.kafka.listeners[0].type to loadbalancer

apiVersion: kafka.strimzi.io/v1beta2
kind: Kafka
metadata:
  name: cluster-a
spec:
  kafka:
    version: 3.6.1
    replicas: 1
    listeners:
      - name: tls
        port: 9093
#        type: internal
        type: loadbalancer
        tls: true
        authentication:
          type: tls
    authorization:
      type: simple
    config:
      offsets.topic.replication.factor: 1
      transaction.state.log.replication.factor: 1
      transaction.state.log.min.isr: 1
      default.replication.factor: 1
      min.insync.replicas: 1
      inter.broker.protocol.version: "3.6"
    storage:
      class: local-path
      size: 1Gi
      type: persistent-claim
  zookeeper:
    replicas: 1
    storage:
      class: local-path
      size: 1Gi
      type: persistent-claim
  entityOperator:
    topicOperator: {}
    userOperator: {}
---

apiVersion: kafka.strimzi.io/v1beta2
kind: Kafka
metadata:
  name: cluster-b
spec:
  kafka:
    version: 3.6.1
    replicas: 1
    listeners:
      - name: tls
        port: 9093
#        type: internal
        type: loadbalancer
        tls: true
        authentication:
          type: tls
    authorization:
      type: simple
    config:
      offsets.topic.replication.factor: 1
      transaction.state.log.replication.factor: 1
      transaction.state.log.min.isr: 1
      default.replication.factor: 1
      min.insync.replicas: 1
      inter.broker.protocol.version: "3.6"
    storage:
      class: local-path
      size: 1Gi
      type: persistent-claim
  zookeeper:
    replicas: 1
    storage:
      class: local-path
      size: 1Gi
      type: persistent-claim
  entityOperator:
    userOperator: {}
---

apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaUser
metadata:
  name: cluster-a-user
  labels:
    strimzi.io/cluster: cluster-a
spec:
  authentication:
    type: tls
  authorization:
    type: simple
    acls:
      # MirrorSourceConnector
      - resource: # Not needed if offset-syncs.topic.location=target
          type: topic
          name: mm2-offset-syncs.cluster-b.internal
        operations:
          - Read
          - DescribeConfigs
          - Write
          - Create
      - resource: # Needed for every topic which is mirrored
          type: topic
          name: "*"
        operations:
          - Read
          - DescribeConfigs
      # MirrorCheckpointConnector
      - resource:
          type: cluster
        operations:
          - Describe
      - resource: # Needed for every group for which offsets are synced
          type: group
          name: "*"
        operations:
          - Describe
---

apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaUser
metadata:
  name: cluster-b-user
  labels:
    strimzi.io/cluster: cluster-b
spec:
  authentication:
    type: tls
  authorization:
    type: simple
    acls:
      # Underlying Kafka Connects internal topics used to store configuration, offsets or status
      - resource:
          type: group
          name: mirrormaker2-cluster
        operations:
          - Read
      - resource:
          type: topic
          name: mirrormaker2-cluster-configs
        operations:
          - Read
          - Describe
          - DescribeConfigs
          - Write
          - Create
      - resource:
          type: topic
          name: mirrormaker2-cluster-status
        operations:
          - Read
          - Describe
          - DescribeConfigs
          - Write
          - Create
      - resource:
          type: topic
          name: mirrormaker2-cluster-offsets
        operations:
          - Read
          - Describe
          - DescribeConfigs
          - Write
          - Create
      # MirrorSourceConnector
      - resource: # Needed for every topic which is mirrored
          type: topic
          name: "*"
        operations:
          - Read
          - Write
          - AlterConfigs
          - Alter
          - Create
      # MirrorCheckpointConnector
      - resource:
          type: cluster
        operations:
          - Describe
      - resource:
          type: topic
          name: cluster-a.checkpoints.internal
        operations:
          - Describe
          - Write
          - Create
      - resource: # Needed for every group for which the offset is synced
          type: group
          name: "*"
        operations:
          - Read
          - Describe
      # MirrorHeartbeatConnector => needed only if the connector is enabled
      # - resource:
      #     type: topic
      #     name: heartbeats
      #   operations:
      #     - Read
      #     - Create
      #     - Describe
      #     - Write
---

apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaMirrorMaker2
metadata:
  name: my-mirror-maker-2
spec:
  version: 3.6.1
  replicas: 1
  connectCluster: "cluster-b" # Must be the target custer
  clusters:
    - alias: "cluster-a"  # Source cluster
      bootstrapServers: cluster-a-kafka-bootstrap:9093
      tls:
        trustedCertificates:
          - secretName: cluster-a-cluster-ca-cert
            certificate: ca.crt
      authentication:
        type: tls
        certificateAndKey:
          secretName: cluster-a-user
          certificate: user.crt
          key: user.key
    - alias: "cluster-b" # Target cluster
      bootstrapServers: cluster-b-kafka-bootstrap:9093
      tls:
        trustedCertificates:
          - secretName: cluster-b-cluster-ca-cert
            certificate: ca.crt
      authentication:
        type: tls
        certificateAndKey:
          secretName: cluster-b-user
          certificate: user.crt
          key: user.key
      config:
        # -1 means it will use the default replication factor configured in the broker
        config.storage.replication.factor: -1
        offset.storage.replication.factor: -1
        status.storage.replication.factor: -1
  mirrors:
    - sourceCluster: "cluster-a"
      targetCluster: "cluster-b"
      sourceConnector:
        tasksMax: 1
        config:
          # -1 means it will use the default replication factor configured in the broker
          replication.factor: -1
          offset-syncs.topic.replication.factor: -1
          sync.topic.acls.enabled: "false"
          refresh.topics.interval.seconds: 60
          sync.topic.acls.interval.seconds: 60
          sync.topic.configs.interval.seconds: 60
          offset.lag.max: 100
      checkpointConnector:
        tasksMax: 1
        config:
          # -1 means it will use the default replication factor configured in the broker
          checkpoints.topic.replication.factor: -1
          sync.group.offsets.enabled: "true"
          emit.checkpoints.interval.seconds: 60
          sync.group.offsets.interval.seconds: 60
          refresh.groups.interval.seconds: 60
      topicsPattern: ".*"
      groupsPattern: ".*"

the mirrormake2 pod logs:

	at java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2073)
	at org.apache.kafka.common.internals.KafkaFutureImpl.get(KafkaFutureImpl.java:165)
	at org.apache.kafka.connect.runtime.WorkerConfig.lookupKafkaClusterId(WorkerConfig.java:299)
	... 5 more
Caused by: org.apache.kafka.common.errors.SslAuthenticationException: Failed to process post-handshake messages
Caused by: javax.net.ssl.SSLHandshakeException: Received fatal alert: bad_certificate
	at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:131)
	at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:117)
	at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:365)
	at java.base/sun.security.ssl.Alert$AlertConsumer.consume(Alert.java:293)
	at java.base/sun.security.ssl.TransportContext.dispatch(TransportContext.java:204)
	at java.base/sun.security.ssl.SSLTransport.decode(SSLTransport.java:172)
	at java.base/sun.security.ssl.SSLEngineImpl.decode(SSLEngineImpl.java:736)
	at java.base/sun.security.ssl.SSLEngineImpl.readRecord(SSLEngineImpl.java:691)
	at java.base/sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:506)
	at java.base/sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:482)
	at java.base/javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:679)
	at org.apache.kafka.common.network.SslTransportLayer.read(SslTransportLayer.java:576)
	at org.apache.kafka.common.network.NetworkReceive.readFrom(NetworkReceive.java:84)
	at org.apache.kafka.common.network.KafkaChannel.receive(KafkaChannel.java:452)
	at org.apache.kafka.common.network.KafkaChannel.read(KafkaChannel.java:402)
	at org.apache.kafka.common.network.Selector.attemptRead(Selector.java:674)
	at org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:576)
	at org.apache.kafka.common.network.Selector.poll(Selector.java:481)
	at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:571)
	at org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.processRequests(KafkaAdminClient.java:1381)
	at org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.run(KafkaAdminClient.java:1312)
	at java.base/java.lang.Thread.run(Thread.java:840)

ths svc:

cluster-a-kafka-bootstrap            ClusterIP      10.97.62.77      <none>         9091/TCP                                                                                                                   7h15m
cluster-a-kafka-brokers              ClusterIP      None             <none>         9090/TCP,9091/TCP,8443/TCP                                                                                                 7h15m
cluster-a-kafka-tls-0                LoadBalancer   10.103.28.138    10.6.178.183   9093:30268/TCP                                                                                                             7h15m
cluster-a-kafka-tls-bootstrap        LoadBalancer   10.100.29.121    10.6.178.182   9093:32493/TCP                                                                                                             7h15m
cluster-a-zookeeper-client           ClusterIP      10.110.249.34    <none>         2181/TCP                                                                                                                   7h15m
cluster-a-zookeeper-nodes            ClusterIP      None             <none>         2181/TCP,2888/TCP,3888/TCP                                                                                                 7h15m
cluster-b-kafka-bootstrap            ClusterIP      10.103.51.250    <none>         9091/TCP                                                                                                                   7h17m
cluster-b-kafka-brokers              ClusterIP      None             <none>         9090/TCP,9091/TCP,8443/TCP                                                                                                 7h17m
cluster-b-kafka-tls-0                LoadBalancer   10.110.185.151   10.6.178.181   9093:32360/TCP                                                                                                             7h17m
cluster-b-kafka-tls-bootstrap        LoadBalancer   10.110.44.42     10.6.178.180   9093:30117/TCP                                                                                                             7h17m
cluster-b-zookeeper-client           ClusterIP      10.98.182.185    <none>         2181/TCP                                                                                                                   7h18m
cluster-b-zookeeper-nodes            ClusterIP      None             <none>         2181/TCP,2888/TCP,3888/TCP                                                                                                 7h18m          
my-mirror-maker-2-mirrormaker2       ClusterIP      None             <none>         8083/TCP                                                                                                                   7h18m
my-mirror-maker-2-mirrormaker2-api   ClusterIP      10.96.151.133    <none>         8083/TCP                                                                                                                   7h18m

I think the spec.clusters[0].bootstrapServers and spec.clusters[1].bootstrapServers may be no right, the port of svc cluster-b-kafka-bootstrap is 9091,

kind: KafkaMirrorMaker2
metadata:
  name: my-mirror-maker-2
spec:
  version: 3.6.1
  replicas: 1
  connectCluster: "cluster-b" # Must be the target custer
  clusters:
    - alias: "cluster-a"  # Source cluster
      bootstrapServers: cluster-a-kafka-bootstrap:9093
       # ...
    - alias: "cluster-b" # Target cluster
      bootstrapServers: cluster-b-kafka-bootstrap:9093

if I change to cluster-b-kafka-tls-bootstrap:9093,it now work either, I noticed this issue: #7713, I decode the two cluster secrets，no tls svc in it:

kubectl get secret cluster-b-kafka-brokers -o jsonpath='{.data.cluster-b-kafka-0\.crt}' | base64 -d |openssl x509 -noout -text| grep DNS | tr , '\n' | cut -d: -f2 
cluster-b-kafka-0.cluster-b-kafka-brokers.m0103.svc
cluster-b-kafka-bootstrap
cluster-b-kafka-brokers
cluster-b-kafka-brokers.m0103.svc
cluster-b-kafka-brokers.m0103
10.6.178.181
cluster-b-kafka-bootstrap.m0103
10.6.178.180
cluster-b-kafka-bootstrap.m0103.svc.cluster.local
cluster-b-kafka-brokers.m0103.svc.cluster.local
cluster-b-kafka-bootstrap.m0103.svc
cluster-b-kafka-0.cluster-b-kafka-brokers.m0103.svc.cluster.local

kubectl get secret cluster-a-kafka-brokers -o jsonpath='{.data.cluster-a-kafka-0\.crt}' | base64 -d |openssl x509 -noout -text| grep DNS | tr , '\n' | cut -d: -f2 
cluster-a-kafka-brokers.m0103
cluster-a-kafka-brokers.m0103.svc
cluster-a-kafka-bootstrap.m0103.svc.cluster.local
cluster-a-kafka-brokers.m0103.svc.cluster.local
cluster-a-kafka-0.cluster-a-kafka-brokers.m0103.svc
10.6.178.183
cluster-a-kafka-bootstrap.m0103
10.6.178.182
cluster-a-kafka-brokers
cluster-a-kafka-bootstrap.m0103.svc
cluster-a-kafka-0.cluster-a-kafka-brokers.m0103.svc.cluster.local
cluster-a-kafka-bootstrap

I stucked at this, it is a bug when user loadbalancer in this example or I missed something?

Steps to reproduce

No response

Expected behavior

No response

Strimzi version

0.1.1

Kubernetes version

v1.26.5

Installation method

helm install

Infrastructure

Bare-metal

Configuration files and logs

No response

Additional context

No response

scholzj · 2024-03-22T07:09:41Z

scholzj
Mar 22, 2024
Maintainer

If you badly adapt an example, that does not make it a bug, so please don't it as a bug! The issue template states so quite clearly!

Also, you cannot make up the bootstrap addresses any way you want or pick up a random port and expect it to work. If you use a load balancer listener, you have to use the load balancer as a bootstrap address. You can always find the right bootstrap address in the .status section of the Kafka custom resource once the cluster is deployed.

4 replies

usernameisnull Mar 22, 2024
Author

changed storage to local-path,
add some configs
change some configs
change internal to loadbalancer

didn't change the bootstrapServers field, and no such svc with the port in original example file

clusters:
    - alias: "cluster-a"  # Source cluster
      bootstrapServers: cluster-a-kafka-bootstrap:9093

or

alias: "cluster-b" # Target cluster
      bootstrapServers: cluster-b-kafka-bootstrap:9093

list svc

cluster-a-kafka-bootstrap            ClusterIP      10.97.62.77      <none>         9091/TCP                                                                                                                   7h15m
cluster-a-kafka-brokers              ClusterIP      None             <none>         9090/TCP,9091/TCP,8443/TCP                                                                                                 7h15m
cluster-a-kafka-tls-0                LoadBalancer   10.103.28.138    10.6.178.183   9093:30268/TCP                                                                                                             7h15m
cluster-a-kafka-tls-bootstrap        LoadBalancer   10.100.29.121    10.6.178.182   9093:32493/TCP                                                                                                             7h15m
cluster-a-zookeeper-client           ClusterIP      10.110.249.34    <none>         2181/TCP                                                                                                                   7h15m
cluster-a-zookeeper-nodes            ClusterIP      None             <none>         2181/TCP,2888/TCP,3888/TCP                                                                                                 7h15m
cluster-b-kafka-bootstrap            ClusterIP      10.103.51.250    <none>         9091/TCP                                                                                                                   7h17m
cluster-b-kafka-brokers              ClusterIP      None             <none>         9090/TCP,9091/TCP,8443/TCP                                                                                                 7h17m
cluster-b-kafka-tls-0                LoadBalancer   10.110.185.151   10.6.178.181   9093:32360/TCP                                                                                                             7h17m
cluster-b-kafka-tls-bootstrap        LoadBalancer   10.110.44.42     10.6.178.180   9093:30117/TCP                                                                                                             7h17m
cluster-b-zookeeper-client           ClusterIP      10.98.182.185    <none>         2181/TCP                                                                                                                   7h18m
cluster-b-zookeeper-nodes            ClusterIP      None             <none>         2181/TCP,2888/TCP,3888/TCP                                                                                                 7h18m          
my-mirror-maker-2-mirrormaker2       ClusterIP      None             <none>         8083/TCP                                                                                                                   7h18m
my-mirror-maker-2-mirrormaker2-api   ClusterIP      10.96.151.133    <none>         8083/TCP

scholzj Mar 22, 2024
Maintainer

You changed the listener type, so you have to change everything related to it including the bootstrap addresses that are based on it. Maybe you should actually read the docs or try the examples as they are before running around and changing things without understanding them.

usernameisnull Mar 22, 2024
Author

ok, got it. I tried cluster-b-kafka-tls-bootstrap:9093 as the bootstrap address then failed. The status shows, should use 10.6.178.180:9093 as the bootstrap, I use this address, it's ok now, thanks

scholzj Mar 22, 2024
Maintainer

Great, glad it works now.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Strimzi

[Bug]: kafka use loadbalancer in kmm2 tls example not work #9864

Uh oh!

{{title}}

Uh oh!

Replies: 1 comment 4 replies

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{editor}}'s edit

{{editor}}'s edit

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Select a reply

Uh oh!

Strimzi

[Bug]: kafka use loadbalancer in kmm2 tls example not work #9864

Uh oh!

usernameisnull Mar 22, 2024

Bug Description

Steps to reproduce

Expected behavior

Strimzi version

Kubernetes version

Installation method

Infrastructure

Configuration files and logs

Additional context

Replies: 1 comment · 4 replies

Uh oh!

scholzj Mar 22, 2024 Maintainer

Uh oh!

usernameisnull Mar 22, 2024 Author

Uh oh!

Uh oh!

scholzj Mar 22, 2024 Maintainer

Uh oh!

usernameisnull Mar 22, 2024 Author

Uh oh!

scholzj Mar 22, 2024 Maintainer

usernameisnull
Mar 22, 2024

Replies: 1 comment 4 replies

scholzj
Mar 22, 2024
Maintainer

usernameisnull Mar 22, 2024
Author

scholzj Mar 22, 2024
Maintainer

usernameisnull Mar 22, 2024
Author

scholzj Mar 22, 2024
Maintainer