Cannot start the mongodb-source-connector

[miracle0930]

Hi! I am deploying mongo-kafka-connector by using strimzi. But I was blocked by this error:

2024-01-25 12:56:24,254 INFO 172.17.0.1 - - [25/Jan/2024:12:56:24 +0000] "GET /connectors/mongodb-source-connector/config HTTP/1.1" 404 75 "-" "-" 7 (org.apache.kafka.connect.runtime.rest.RestServer) [qtp542766981-24]
2024-01-25 12:56:24,277 ERROR Uncaught exception in REST call to /connectors/mongodb-source-connector/config (org.apache.kafka.connect.runtime.rest.errors.ConnectExceptionMapper) [qtp542766981-23]
org.apache.kafka.connect.errors.ConnectException: Failed to find any class that implements Connector and which name matches com.mongodb.kafka.connect.MongoSourceConnector, available connectors are: PluginDesc{klass=class org.apache.kafka.connect.mirror.MirrorCheckpointConnector, name='org.apache.kafka.connect.mirror.MirrorCheckpointConnector', version='3.6.1', encodedVersion=3.6.1, type=source, typeName='source', location='classpath'}, PluginDesc{klass=class org.apache.kafka.connect.mirror.MirrorHeartbeatConnector, name='org.apache.kafka.connect.mirror.MirrorHeartbeatConnector', version='3.6.1', encodedVersion=3.6.1, type=source, typeName='source', location='classpath'}, PluginDesc{klass=class org.apache.kafka.connect.mirror.MirrorSourceConnector, name='org.apache.kafka.connect.mirror.MirrorSourceConnector', version='3.6.1', encodedVersion=3.6.1, type=source, typeName='source', location='classpath'}
	at org.apache.kafka.connect.runtime.isolation.Plugins.connectorClass(Plugins.java:319)
	at org.apache.kafka.connect.runtime.isolation.Plugins.newConnector(Plugins.java:290)
	at org.apache.kafka.connect.runtime.AbstractHerder.lambda$getConnector$6(AbstractHerder.java:702)
	at java.base/java.util.concurrent.ConcurrentHashMap.computeIfAbsent(ConcurrentHashMap.java:1708)
	at org.apache.kafka.connect.runtime.AbstractHerder.getConnector(AbstractHerder.java:702)
	at org.apache.kafka.connect.runtime.AbstractHerder.validateConnectorConfig(AbstractHerder.java:470)
	at org.apache.kafka.connect.runtime.AbstractHerder.lambda$validateConnectorConfig$2(AbstractHerder.java:390)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
	at java.base/java.lang.Thread.run(Thread.java:840)

This is my configuration file for kafka connect:

---
apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaConnect
metadata:
  name: kafka-cluster
  namespace: ns
  annotations:
    strimzi.io/use-connector-resources: "true"
spec:
  version: 3.6.1
  replicas: 1
  bootstrapServers: kafka:9092
  config:
    config.storage.replication.factor: 1
    offset.storage.replication.factor: 1
    status.storage.replication.factor: 1
    key.converter": org.apache.kafka.connect.json.JsonConverter
    key.converter.schemas.enable: false
    value.converter: org.apache.kafka.connect.json.JsonConverter
    value.converter.schemas.enable: false

and this is the configuration file for connector

apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaConnector
metadata:
  name: mongodb-source-connector
  namespace: ns
  labels:
    strimzi.io/cluster: kafka-cluster
spec:
  class: com.mongodb.kafka.connect.MongoSourceConnector
  tasksMax: 2
  config:
    connection.uri: mongodb://mongo:27017/
    pipeline: []
    change.stream.full.document: updateLookup
    output.format.key: json
    output.format.value: json
    output.json.formatter: com.mongodb.kafka.connect.source.json.formatter.SimplifiedJson

It looks like the MongoSourceConnector is not being installed. Can I get some help here? Thanks!

[scholzj]

I guess that suggests there is no MongoDB connector in the container image you use. So, how did you add the connector to it?

[miracle0930]

Thanks for your answer! Yes, I think the error is pretty straightforward. That's kind of my second question. I am quite new in this area, so to add a connector based on my understanding there are two ways:

• add a docker image URL to spec.image field
• add a spec.build section to create a custom image

Is this correct?

[scholzj]

That is correct, yes. For the first case, you can basically write a Dockerfile that starts from the Strimzi container image as a base image and adds the connector(s) you need. There is an example in the docs. You can then build it locally or in some CI, push it to registry such as DockerHub or Quay and specify it in .spec.image. Or you can use the .spec.build feature -> that does the build for you, but you need to point it to a container registry where it can push the container image.

[miracle0930]

Thanks, to create a custom image, I saw there is pushSecret field, what should be the value for it, something like identity token? I tried by using the .spec.build earlier but it gives me this error:

MountVolume.SetUp failed for volume "docker-credentials" : secret "xxxxxx" not found

[scholzj]

You have to link a secret with the container registry credentials. You should be able to do it for example using one of the ways described here: https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/#registry-secret-existing-credentials or here: https://kubernetes.io/docs/concepts/configuration/secret/#docker-config-secrets

[miracle0930]

Thanks!