Strimzi
Kafka Connect unable to write to offsets topics when using Confluent Cloud #8341
-
|
I'm running a KafkaConnect cluster with the following specification - apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaConnect
metadata:
name: connectcluster
namespace: kafka
annotations:
strimzi.io/use-connector-resources: "true"
spec:
version: 3.3.2
replicas: 1
bootstrapServers: pkc-xxxx.us-west4.gcp.confluent.cloud:9092
authentication:
type: plain
username: XXXXXXXX
passwordSecret:
secretName: brokerpass
password: password
tls:
trustedCertificates: []
externalConfiguration:
image: avinashupadhya99/strimzi-jdbc-connect:1.0.0
metricsConfig:
type: jmxPrometheusExporter
valueFrom:
configMapKeyRef:
name: prometh-config-map
key: prometh-key
logging:
type: inline
loggers:
log4j.rootLogger: "TRACE"
config:
group.id: connect-strimzi
offset.storage.topic: _offsets-sb
config.storage.topic: _configs-sb
status.storage.topic: _status-sb
# -1 means it will use the default replication factor configured in the broker
config.storage.replication.factor: -1
offset.storage.replication.factor: -1
status.storage.replication.factor: -1
fetch.max.wait.ms: 2000
max.poll.records: 1The Connect cluster is not able to fetch from the offsets topic and sometimes, the status topic. I see the following in the logs - I have tried with both the Strimzi Kafka image and Confluent Kafka image. Both result in the same situation. |
Beta Was this translation helpful? Give feedback.
Replies: 2 comments 2 replies
-
Not sure what exactly this means -> you can use only container images based on our images with Strimzi. As for the error, maybe its more a question on whoever runs the cluster? Also, its TRACE message -> so one has to wonder if this is actually something to be concerned about. |
Beta Was this translation helpful? Give feedback.
-
|
I switched to using According to the Confluent documentation, the JDBC connector does not need any additional drivers for postgres. When I query the connect REST API, I am able to see the connector configured but it does not work. GET /connector-plugins [
{
"class": "io.confluent.connect.jdbc.JdbcSinkConnector",
"type": "sink",
"version": "10.6.1"
},
{
"class": "io.confluent.connect.jdbc.JdbcSourceConnector",
"type": "source",
"version": "10.6.1"
},
{
"class": "org.apache.kafka.connect.mirror.MirrorCheckpointConnector",
"type": "source",
"version": "3.3.1"
},
{
"class": "org.apache.kafka.connect.mirror.MirrorHeartbeatConnector",
"type": "source",
"version": "3.3.1"
},
{
"class": "org.apache.kafka.connect.mirror.MirrorSourceConnector",
"type": "source",
"version": "3.3.1"
}
]I even tried adding the Postgres JDBC driver jar into |
Beta Was this translation helpful? Give feedback.
-
|
I guess the plugin needs to be part of the JDBC connector -> in the same subdirectory. Please check the doce about the structure inside the plugins directory. |
Beta Was this translation helpful? Give feedback.
-
|
The issue was that it was missing other necessary JAR files in the plugins directory. |
Beta Was this translation helpful? Give feedback.
I guess the plugin needs to be part of the JDBC connector -> in the same subdirectory. Please check the doce about the structure inside the plugins directory.