Failed to connect to and describe Kafka cluster. Check worker's broker connection and security properties

[eyanez111]

Hello I have a connection issue. My MorrorMaker was working. I added the TLS and then stopped working. I followed all the documentation and the git examples but I am getting this errors:

WARN [AdminClient clientId=adminclient-1] Metadata update failed due to authentication error (org.apache.kafka.clients.admin.internals.AdminMetadataManager) [kafka-admin-client-thread | adminclient-1]
org.apache.kafka.common.errors.SslAuthenticationException: Failed to process post-handshake messages
org.apache.kafka.common.errors.TimeoutException: The AdminClient thread has exited. Call: fetchMetadata
org.apache.kafka.common.errors.TimeoutException: The AdminClient thread has exited. Call: fetchMetadata
2022-03-06 19:05:50,536 ERROR Stopping due to error (org.apache.kafka.connect.cli.ConnectDistributed) [main]
org.apache.kafka.connect.errors.ConnectException: Failed to connect to and describe Kafka cluster. Check worker's broker connection and security properties.
Caused by: java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.SslAuthenticationException: Failed to process post-handshake messages
Caused by: org.apache.kafka.common.errors.SslAuthenticationException: Failed to process post-handshake messages

I did copied the cluster-ca-cert build a secret on the sending side of the source, enable tls authentication on both sides. This is my MMK config:

apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaMirrorMaker2
metadata:
  name: data-sync
spec:
  version: 3.0.0
  replicas: 1
  connectCluster: "target-cluster"
  clusters:
  - alias: "origin-cluster"
    bootstrapServers: 172.20.11.198:9094
  #  tls:
  #    trustedCertificates:
  #      - secretName: origin-cluster-secret
  #        certificate: ca.crt
  - alias: "target-cluster"
    bootstrapServers: 172.21.140.199:9094
    tls:
      trustedCertificates:
        - secretName: target-test-cluster-secret
          certificate: ca.crt
    config:
      # -1 means it will use the default replication factor configured in the broker
      config.storage.replication.factor: 3
      offset.storage.replication.factor: 3
      status.storage.replication.factor: 3

  mirrors:
  - sourceCluster: "my-source-cluster"
    targetCluster: "my-target-cluster"
    sourceConnector:
      config:
        replication.factor: 3
        offset-syncs.topic.replication.factor: 3
        sync.topic.acls.enabled: "false"

    heartbeatConnector:
      config:
        heartbeats.topic.replication.factor: 3


    checkpointConnector:
      config:
        checkpoints.topic.replication.factor: 3
        refresh.groups.interval.seconds: 600
    topicsPattern: ".*"
    groupsPattern: ".*"

and here where I think is the problem (in the strimzi config). I do have the same config on both clusters:
apiVersion: kafka.strimzi.io/v1beta2

kind: Kafka
metadata:
  name: my-cluster
spec:
  kafka:
    version: 3.0.0
    replicas: 3
    listeners:
      - name: plain
        port: 9092
        type: internal
        tls: false
      - name: tls
        port: 9093
        type: internal
        tls: false
      - name: external
        port: 9094
        type: loadbalancer
        tls: true
        authentication:
          type: tls 
    config:
      retention.ms: 604800000
      min.insync.replicas: 2
      offsets.topic.replication.factor: 3
      transaction.state.log.replication.factor: 3
      transaction.state.log.min.isr: 2
      log.message.format.version: "3.0.0"
      inter.broker.protocol.version: "3.0.0"
    resources:
      requests:
        memory: 2309Mi
        cpu: 1000m
      limits:
        memory: 2309Mi
        cpu: 1000m      
    storage:
      type: jbod
      volumes:
      - id: 0
        type: persistent-claim
        size: 2300Gi
        deleteClaim: false
  zookeeper:      
    replicas: 3
    resources:
      requests:
        memory: 1Gi
        cpu: 100m
      limits:
        memory: 1.5Gi
        cpu: 1000m
    storage:
      type: persistent-claim
      size: 100Gi
      deleteClaim: false
  entityOperator:
    topicOperator: {}
    userOperator: {}

I am not sure (since I just need authentication in the target if I should have the:
listeners:

      - name: plain
        port: 9092
        type: internal
        tls: false
      - name: tls
        port: 9093
        type: internal
        tls: false
      - name: external
        port: 9094
        type: loadbalancer
        tls: true
        authentication:
          type: tls

Running on both clusters

Thanks ahead for any help or time that this team dedicates to solve our tech problems

[AAR256]

I'm also facing the same issue here. Any input and help here would be appreciated.

[eyanez111]

The problem here and there is not actual tutorial that explains that is that once you turn on authentication tls in your strimzi configuration, you also have to do a tls authentication on the mirrormaker side. To do that you not just need the ca.crt from the cluster-ca-cert secret but also you need to create a kafkauser then use that user secret to create a secret where you put the user.crt and user.key that you will get from the kafkauser and put it on the other cluster so they authenticate both ways. I added a few examples on my github. hope that helps:
https://github.com/eyanez111/mirrormaker

[AAR256]

Thanks a lot for the above information. That helped a lot. I'm able to get the MirrorMaker in running state. Although I'm do getting this particular error. -

2022-03-15 11:00:44,817 WARN [Producer clientId=producer-4] Got error produce response with correlation id 532 on topic-partition mm2-offset-syncs.cdt-west-2.internal-0, retrying (2147483440 attempts left). Error: NOT_ENOUGH_REPLICAS (org.apache.kafka.clients.producer.internals.Sender) [kafka-producer-network-thread | producer-4]

Please do let me know if you've faced this issue. I have tried increasing the min.insync.replicas for my Kafka Clusters as well. That didn't helped.

[eyanez111]

the replicas need to be the configure to be the same as in your Origin Broker, if you have less you will get this error. Also if in the target you have that replica (try to name it differently) , it can create conflicts. This problem I have seen it a few times and it has been solve most of the times by matching in MMK to have the same replicas than what the origin broker has but once it was a problem on the target... that one I had to delete all the replicas and make sure that my kafkatopics were able to match in terms of replicas, the same I had on the target.