Failed to connect to and describe Kafka cluster on existing Confluent Cloud kafka cluster while deploying kafka connect cluster on kubernetes

[Eliah7]

I tried using the Strimzi KafkaConnect Operator to connect to an existing kafka cluster on Confluent Cloud.

Steps to reproduce the behavior:

1. Install the Strimzi cluster operator chart
   helm repo add strimzi https://strimzi.io/charts/ and helm install strimzi-kafka strimzi/strimzi-kafka-operator
2. Create the secret with username and password from confluent cloud
3. Apply the KafkaConnect resource file to the kubernetes cluster

apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaConnect
metadata:
  name: my-connect-cluster
  annotations:
    strimzi.io/use-connector-resources: "true"
spec:
  version: 2.8.0
  replicas: 1
  authentication: 
    type: plain
    username: ************
    passwordSecret:
      secretName: connect-secret
      password: connect-secret-password
  bootstrapServers: **********************************:9092
  tls:
    trustedCertificates:
      []
  config:
    group.id: connect-cluster
    security.protocol: SASL_PLAIN
    offset.storage.topic: connect-cluster-offsets
    config.storage.topic: connect-cluster-configs
    status.storage.topic: connect-cluster-status
    rest.port: 8083
    rest.advertised.host.name: 0.0.0.0
    rest.advertised.port: 8083
    kafka.security.protocol: SASL_PLAIN
    ssl.endpoint.identification.algorithm: "https"
    config.storage.replication.factor: -1
    offset.storage.replication.factor: -1
    status.storage.replication.factor: -1

3. See error

2021-10-20 09:34:35,748 WARN The configuration 'key.converter' was supplied but isn't a known config. (org.apache.kafka.clients.admin.AdminClientConfig) [main]
2021-10-20 09:34:35,748 WARN The configuration 'consumer.sasl.jaas.config' was supplied but isn't a known config. (org.apache.kafka.clients.admin.AdminClientConfig) [main]
2021-10-20 09:34:35,748 WARN The configuration 'config.storage.topic' was supplied but isn't a known config. (org.apache.kafka.clients.admin.AdminClientConfig) [main]
2021-10-20 09:34:35,749 WARN The configuration 'producer.security.protocol' was supplied but isn't a known config. (org.apache.kafka.clients.admin.AdminClientConfig) [main]
2021-10-20 09:34:35,749 WARN The configuration 'rest.advertised.host.name' was supplied but isn't a known config. (org.apache.kafka.clients.admin.AdminClientConfig) [main]
2021-10-20 09:34:35,749 WARN The configuration 'status.storage.topic' was supplied but isn't a known config. (org.apache.kafka.clients.admin.AdminClientConfig) [main]
2021-10-20 09:34:35,749 WARN The configuration 'client.rack' was supplied but isn't a known config. (org.apache.kafka.clients.admin.AdminClientConfig) [main]
2021-10-20 09:34:35,749 WARN The configuration 'producer.sasl.mechanism' was supplied but isn't a known config. (org.apache.kafka.clients.admin.AdminClientConfig) [main]
2021-10-20 09:34:35,749 WARN The configuration 'rest.port' was supplied but isn't a known config. (org.apache.kafka.clients.admin.AdminClientConfig) [main]
2021-10-20 09:34:35,749 WARN The configuration 'config.storage.replication.factor' was supplied but isn't a known config. (org.apache.kafka.clients.admin.AdminClientConfig) [main]
2021-10-20 09:34:35,749 WARN The configuration 'offset.storage.replication.factor' was supplied but isn't a known config. (org.apache.kafka.clients.admin.AdminClientConfig) [main]
2021-10-20 09:34:35,749 WARN The configuration 'admin.sasl.jaas.config' was supplied but isn't a known config. (org.apache.kafka.clients.admin.AdminClientConfig) [main]
2021-10-20 09:34:35,750 INFO Kafka version: 2.8.0 (org.apache.kafka.common.utils.AppInfoParser) [main]
2021-10-20 09:34:35,750 INFO Kafka commitId: ebb1d6e21cc92130 (org.apache.kafka.common.utils.AppInfoParser) [main]
2021-10-20 09:34:35,751 INFO Kafka startTimeMs: 1634722475749 (org.apache.kafka.common.utils.AppInfoParser) [main]
2021-10-20 09:34:47,540 INFO [AdminClient clientId=adminclient-1] Failed authentication with pkc-lq8v7.eu-central-1.aws.confluent.cloud/3.68.18.251 (Authentication failed) (org.apache.kafka.common.network.Selector) [kafka-admin-client-thread | adminclient-1]
2021-10-20 09:34:47,708 ERROR [AdminClient clientId=adminclient-1] Connection to node -1 (pkc-lq8v7.eu-central-1.aws.confluent.cloud/3.68.18.251:9092) failed authentication due to: Authentication failed (org.apache.kafka.clients.NetworkClient) [kafka-admin-client-thread | adminclient-1]
2021-10-20 09:34:47,718 WARN [AdminClient clientId=adminclient-1] Metadata update failed due to authentication error (org.apache.kafka.clients.admin.internals.AdminMetadataManager) [kafka-admin-client-thread | adminclient-1]
org.apache.kafka.common.errors.SaslAuthenticationException: Authentication failed
2021-10-20 09:34:47,827 INFO App info kafka.admin.client for adminclient-1 unregistered (org.apache.kafka.common.utils.AppInfoParser) [kafka-admin-client-thread | adminclient-1]
2021-10-20 09:34:47,849 INFO [AdminClient clientId=adminclient-1] Metadata update failed (org.apache.kafka.clients.admin.internals.AdminMetadataManager) [kafka-admin-client-thread | adminclient-1]
org.apache.kafka.common.errors.TimeoutException: Call(callName=fetchMetadata, deadlineMs=1634722505760, tries=1, nextAllowedTryMs=-9223372036854775709) timed out at 9223372036854775807 after 1 attempt(s)
Caused by: org.apache.kafka.common.errors.TimeoutException: The AdminClient thread has exited. Call: fetchMetadata
2021-10-20 09:34:47,849 INFO [AdminClient clientId=adminclient-1] Metadata update failed (org.apache.kafka.clients.admin.internals.AdminMetadataManager) [kafka-admin-client-thread | adminclient-1]
org.apache.kafka.common.errors.TimeoutException: Call(callName=fetchMetadata, deadlineMs=1634722517744, tries=1, nextAllowedTryMs=-9223372036854775709) timed out at 9223372036854775807 after 1 attempt(s)
Caused by: org.apache.kafka.common.errors.TimeoutException: The AdminClient thread has exited. Call: fetchMetadata
2021-10-20 09:34:49,243 INFO Metrics scheduler closed (org.apache.kafka.common.metrics.Metrics) [kafka-admin-client-thread | adminclient-1]
2021-10-20 09:34:49,262 INFO Closing reporter org.apache.kafka.common.metrics.JmxReporter (org.apache.kafka.common.metrics.Metrics) [kafka-admin-client-thread | adminclient-1]
2021-10-20 09:34:49,264 INFO Metrics reporters closed (org.apache.kafka.common.metrics.Metrics) [kafka-admin-client-thread | adminclient-1]
2021-10-20 09:34:49,274 ERROR Stopping due to error (org.apache.kafka.connect.cli.ConnectDistributed) [main]
org.apache.kafka.connect.errors.ConnectException: Failed to connect to and describe Kafka cluster. Check worker's broker connection and security properties.
at org.apache.kafka.connect.util.ConnectUtils.lookupKafkaClusterId(ConnectUtils.java:70)
	at org.apache.kafka.connect.util.ConnectUtils.lookupKafkaClusterId(ConnectUtils.java:51)
	at org.apache.kafka.connect.cli.ConnectDistributed.startConnect(ConnectDistributed.java:97)
	at org.apache.kafka.connect.cli.ConnectDistributed.main(ConnectDistributed.java:80)
Caused by: java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.SaslAuthenticationException: Authentication failed
	at org.apache.kafka.common.internals.KafkaFutureImpl.wrapAndThrow(KafkaFutureImpl.java:45)
	at org.apache.kafka.common.internals.KafkaFutureImpl.access$000(KafkaFutureImpl.java:32)
	at org.apache.kafka.common.internals.KafkaFutureImpl$SingleWaiter.await(KafkaFutureImpl.java:89)
	at org.apache.kafka.common.internals.KafkaFutureImpl.get(KafkaFutureImpl.java:260)
	at org.apache.kafka.connect.util.ConnectUtils.lookupKafkaClusterId(ConnectUtils.java:64)
	... 3 more
Caused by: org.apache.kafka.common.errors.SaslAuthenticationException: Authentication failed

Expected behavior
I expected a kafka connect cluster of one pod to connect to the confluent cloud kafka cluster successfully.

Environment (please complete the following information):

• Strimzi version: strimzi/operator:0.26.0
• Installation method: Yaml files and Helm
• Kubernetes cluster: Kubernetes v1.17.0 on Docker '19.03.5'
• Infrastructure: Minikube

[scholzj]

You seem to have a lot of invalid configuration options in the .spec.config section. But they should be just ignored I guess. The error suggests that it failed to authenticate => so you should probably double check that you have all the settings right, that the password is properly set in the secret, that the server is actually using a TLS with some certificate signed by a public CA etc.

[tooptoop4]

@Eliah7 did u solve?

[ggnanasekaran77]

I too facing the same problem
@Eliah7 Did you solve this is issue?