No default docker image for Kafka Connect?

[leandroblanc]

Hello, I'm starting the usage of Strimzi (newbie here).

I have deployed the Strimzi operator in Kubernetes. I have defined a Kafka cluster and it worked ("apiVersion: kafka.strimzi.io/v1beta2" -> "kind: Kafka"). So far so good.

Now I'm trying to deploy Kafka Connect, so I'm creating the CRD:

apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaConnect
metadata:
  name: kafka-connect
  annotations:
    strimzi.io/use-connector-resources: "true"
spec:
    image:                <--- WHAT TO PUT HERE?
    ....

My question is: what's the docker image I need to use? I found https://hub.docker.com/r/strimzi/kafka-connect but latest image was built 3 years ago. I found a Strimzi account in Quay.io but I can't see Kafka Connect image there.

Am I forced to build my own Kafka Connect image? If yes, can you please point me to the documentation that explains why? I think I'm missing something.

Thank you very much in advance!

[scholzj]

Our images are in Quay.io, not DockerHub (we moved ling time ago after Docker Hub introduced different service limits for free accounts): https://quay.io/organization/strimzi

the image field is optional. So you do not have to specify it. If you don't, the default image will be this one: https://quay.io/repository/strimzi/kafka ... it will deploy Connect, but basically without any connectors. So it might not be much use for you.

You either have to build your own container image with the connector you want or use the Build feature to have Strimzi build its own for you. This should get you started: https://strimzi.io/docs/operators/latest/full/deploying.html#using-kafka-connect-with-plug-ins-str

[leandroblanc]

@scholzj Hi Jacub, thanks a lot! I have removed the image field and it worked, thanks!

[Koalaelch]

Hey @scholzj
I would like to deploy a debezium-connector on OpenShift. Therefore I need KafkaConnect. I followed these instructions: https://access.redhat.com/documentation/en-us/red_hat_amq/7.3/html/using_amq_streams_on_openshift_container_platform/getting-started-str#using-openshift-s2i-create-image-str noticing that examples/kafka-connect/kafka-connect-s2i.yaml doesn't exist in amq-streams-2.4.0-ocp-install-examples.zip. So I executed oc apply -f examples/connect/kafka-connect.yaml. Unfortunatly, I get a CrashLoopBackOff:

2023-07-27 13:01:39,214 INFO App info kafka.admin.client for adminclient-1 unregistered (org.apache.kafka.common.utils.AppInfoParser) [kafka-admin-client-thread | adminclient-1]
2023-07-27 13:01:39,214 INFO [AdminClient clientId=adminclient-1] Metadata update failed (org.apache.kafka.clients.admin.internals.AdminMetadataManager) [kafka-admin-client-thread | adminclient-1]
org.apache.kafka.common.errors.TimeoutException: The AdminClient thread has exited. Call: fetchMetadata
2023-07-27 13:01:39,214 INFO [AdminClient clientId=adminclient-1] Timed out 1 remaining operation(s) during close. (org.apache.kafka.clients.admin.KafkaAdminClient) [kafka-admin-client-thread | adminclient-1]
2023-07-27 13:01:39,217 INFO Metrics scheduler closed (org.apache.kafka.common.metrics.Metrics) [kafka-admin-client-thread | adminclient-1]
2023-07-27 13:01:39,217 INFO Closing reporter org.apache.kafka.common.metrics.JmxReporter (org.apache.kafka.common.metrics.Metrics) [kafka-admin-client-thread | adminclient-1]
2023-07-27 13:01:39,217 INFO Metrics reporters closed (org.apache.kafka.common.metrics.Metrics) [kafka-admin-client-thread | adminclient-1]
2023-07-27 13:01:39,217 ERROR Stopping due to error (org.apache.kafka.connect.cli.ConnectDistributed) [main]
org.apache.kafka.connect.errors.ConnectException: Failed to connect to and describe Kafka cluster. Check worker's broker connection and security properties.
at org.apache.kafka.connect.runtime.WorkerConfig.lookupKafkaClusterId(WorkerConfig.java:347)
at org.apache.kafka.connect.runtime.WorkerConfig.lookupKafkaClusterId(WorkerConfig.java:327)
at org.apache.kafka.connect.runtime.WorkerConfig.kafkaClusterId(WorkerConfig.java:457)
at org.apache.kafka.connect.cli.ConnectDistributed.startConnect(ConnectDistributed.java:100)
at org.apache.kafka.connect.cli.ConnectDistributed.main(ConnectDistributed.java:83)
Caused by: java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.TimeoutException: Timed out waiting for a node assignment. Call: listNodes
at java.base/java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:396)
at java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2073)
at org.apache.kafka.common.internals.KafkaFutureImpl.get(KafkaFutureImpl.java:165)
at org.apache.kafka.connect.runtime.WorkerConfig.lookupKafkaClusterId(WorkerConfig.java:341)
... 4 more

This is my kafka-connect.yaml:

apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaConnect
metadata:
  name: debezium-connect-cluster
  annotations:
    strimzi.io/use-connector-resources: "true"
spec:
  version: 3.4.0
  replicas: 1
  bootstrapServers: debezium-cluster-kafka-bootstrap:9093
  config:
    config.providers: secrets
    config.providers.secrets.class: io.strimzi.kafka.KubernetesSecretConfigProvider
    group.id: connect-cluster
    offset.storage.topic: connect-cluster-offsets
    config.storage.topic: connect-cluster-configs
    status.storage.topic: connect-cluster-status
    # -1 means it will use the default replication factor configured in the broker
    config.storage.replication.factor: -1
    offset.storage.replication.factor: -1
    status.storage.replication.factor: -1

I'm also a newbie. does my approach look okay at first glance? do i still need to activate CRD? Is the CrashLoopBackOff more related to my OpenShift configuration? I would really appreciate your help.
Many greetings

[scholzj]

Questions and issues with AMQ Streams should be reported to Red Hat Support. AMQ Streams is based on Strimzi, but it is not the same. The documentation you are linking to is also several years old. So it is not clear to me why are you referring to it -> do you actually use that version? If not, you should use the documentation corresponding to the right version of Strimzi which you use. Understanding the version you use is important to point you to the right direction.