Strimzi
-
|
Hello! I've been trying to setup a mongo-db-sink connector in Kubernetes but I am getting some errors. How I did it
Dockerfile: Inside ./connectors I have downloaded The following is my kafka-connect-cluster.yaml apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaConnect
metadata:
name: kafka-connect-cluster
annotations:
strimzi.io/use-connector-resources: "true"
spec:
image: .../strimzi/kafka_connect:3.1.0 # This is the image I built in the step before
template:
pod:
imagePullSecrets:
- name: <hidden>
version: 3.1.0
replicas: 1
bootstrapServers: <hidden>
config:
group.id: connect-cluster
offset.storage.topic: strimzi-connect-cluster-offsets
config.storage.topic: strimzi-connect-cluster-configs
status.storage.topic: strimzi-connect-cluster-statusMy mongodb-sink-connector.yaml apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaConnector
metadata:
name: mongodb-sink-connector
labels:
strimzi.io/cluster: kafka-connect-cluster
spec:
class: com.mongodb.kafka.connect.MongoSinkConnector
tasksMax: 2
config:
topics: beta-topic
connection.uri: mongo.mongo.database.svc.cluster.local/?&replicaSet=rs0
database: kafka
collection: test-backup
post.processor.chain: com.mongodb.kafka.connect.sink.processor.DocumentIdAdder,com.mongodb.kafka.connect.sink.processor.KafkaMetaAdder
key.converter: org.apache.kafka.connect.storage.StringConverter
key.converter.schemas.enable: false
value.converter: org.apache.kafka.connect.json.JsonConverter
value.converter.schemas.enable: falseThis is what I get in my And my apiVersion: kafka.strimzi.io/v1beta2
kind: Kafka
metadata:
name: messaging
spec:
kafka:
version: 3.1.0 #Apache Kafka version
replicas: 3
listeners:
- name: plain
port: 9092
type: internal
tls: false
- name: tls
port: 9093
type: internal
tls: true
- name: external
port: 9094
type: nodeport
tls: false
config:
offsets.topic.replication.factor: 1
transaction.state.log.replication.factor: 1
transaction.state.log.min.isr: 1
default.replication.factor: 1
min.insync.replicas: 1
inter.broker.protocol.version: "3.1"
storage:
type: jbod
volumes:
- id: 0
type: persistent-claim
size: 100Gi
deleteClaim: false
zookeeper:
replicas: 3
storage:
type: persistent-claim
size: 100Gi
deleteClaim: false
entityOperator:
topicOperator: {}
userOperator: {}Any idea of what I am doing wrong? I found this other issue which is similar, and it seems it should be fixed in 0.29 but that's what I am using and I am still hitting the error. Thanks. |
Beta Was this translation helpful? Give feedback.
Answered by
scholzj
Jul 13, 2022
Replies: 1 comment 6 replies
-
|
Without a full logs, it is absolutely not clear what is causing this and where it is happening. The issue you linked is about the Cluster operator but you say your log snippet is from Connect, so it has probably nothing to do with it (+ it is not fixed in 0.29 - it will be fixed only in 0.30.0) |
Beta Was this translation helpful? Give feedback.
6 replies
-
|
Well, the error seems to be quite clear: |
Beta Was this translation helpful? Give feedback.
-
|
Hey @scholzj , I downloaded the connector from Then I created a custom docker image:
This is my dockerfile: FROM quay.io/strimzi/kafka:0.29.0-kafka-3.2.0
USER root:root
COPY connectors /opt/kafka/plugins/
USER 1001The image gets tagged and uploaded to a private registry, and finally used in the kafka-connect.yaml apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaConnect
metadata:
name: kafka-connect-cluster
annotations:
strimzi.io/use-connector-resources: "true"
spec:
image: <hidden>/strimzi/kafka_connect:3.2.0
containers:
imagePullPolicy: "Always"
template:
pod:
imagePullSecrets:
- name: <hidden>
version: 3.1.0
replicas: 1
bootstrapServers: messaging-kafka-bootstrap:9092
config:
group.id: connect-cluster
offset.storage.topic: strimzi-connect-cluster-offsets
config.storage.topic: strimzi-connect-cluster-configs
status.storage.topic: strimzi-connect-cluster-statusAnd finally the mongodb-connector.yaml for k8s apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaConnector
metadata:
name: mongodb-sink-connector
labels:
strimzi.io/cluster: kafka-connect-cluster
spec:
class: com.mongodb.kafka.connect.MongoSinkConnector
tasksMax: 2
config:
topics: beta-topic
connection.uri: mongodev-0.mongo.database.svc.cluster.local:27017,mongodev-1.mongo.database.svc.cluster.local:27017,mongodev-2.mongo.database.svc.cluster.local:27017/?replicaSet=rs0
database: kafka
collection: test-backup
post.processor.chain: com.mongodb.kafka.connect.sink.processor.DocumentIdAdder,com.mongodb.kafka.connect.sink.processor.KafkaMetaAdder
key.converter: org.apache.kafka.connect.storage.StringConverter
key.converter.schemas.enable: false
value.converter: org.apache.kafka.connect.json.JsonConverter
value.converter.schemas.enable: false |
Beta Was this translation helpful? Give feedback.
-
|
I guess the structure should be fine. But I do not know the connector. So I have no idea if the ZIP contains all JARs or if you need to add something more etc. You would probably need to ask the connector authors. |
Beta Was this translation helpful? Give feedback.
-
@scholzj I found out I wasn't updating the image, since the ImagePullPolicy is not set to Always by default. I am now getting an error for the topics field as I am trying to use an array in the mongodb-connector.yaml and Strimzi doesn't really like it. Shouldn't the topic field be an array? As it's also named "topicS" |
Beta Was this translation helpful? Give feedback.
-
|
@scholzj never mind, I was feeding an array while a simple string was fine:
Thank you! |
Beta Was this translation helpful? Give feedback.
Answer selected by
ginwakeup
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Without a full logs, it is absolutely not clear what is causing this and where it is happening. The issue you linked is about the Cluster operator but you say your log snippet is from Connect, so it has probably nothing to do with it (+ it is not fixed in 0.29 - it will be fixed only in 0.30.0)