How can I see the connector log?

[smeeklai]

I'm trying to run s3-sink using the KafkaConnect. I'm seeing the connector status is ready and it seems to run fine but I expect to see some error logs because the connector doesn't have permission to interact with the bucket yet. However, I can't find it in the connect pod. Please guide me.

Kafka cluster: MSK

KafkaConnect

apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaConnect
metadata:
  name: kafka-connect-s3
  namespace: strimzi-kafka-operator
  annotations:
   strimzi.io/use-connector-resources: "true"
spec:
  version: 3.4.0
  replicas: 1
  bootstrapServers: xxx:9096,xxx:9096
  image: xxxx:tmp
  tls:
    trustedCertificates: []
  authentication:
    passwordSecret:
      password: password
      secretName: creds
    type: scram-sha-512
    username: app
  config:
    group.id: connect-cluster
    offset.storage.topic: connect-cluster-offsets
    config.storage.topic: connect-cluster-configs
    status.storage.topic: connect-cluster-status
    # -1 means it will use the default replication factor configured in the broker
    config.storage.replication.factor: -1
    offset.storage.replication.factor: -1
    status.storage.replication.factor: -1

KafkaConnector

apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaConnector
metadata:
  name: kafka-connect-s3-connector
  namespace: strimzi-kafka-operator
  labels:
    strimzi.io/cluster: kafka-connect-s3
spec:
  class: io.confluent.connect.s3.S3SinkConnector
  tasksMax: 1
  config:
    s3.region: ap-southeast-1
    s3.part.retries: 1
    format.class: io.confluent.connect.s3.format.json.JsonFormat
    flush.size: 1
    schema.compatibility: NONE
    topics.regex: ^(?!__)\\S+$
    tasks.max: 2
    partitioner.class: io.confluent.connect.storage.partitioner.DefaultPartitioner
    storage.class: io.confluent.connect.s3.storage.S3Storage
    s3.bucket.name: xxxxx
    errors.log.enable: true

Logs:
kafka-connect-s3-connect.log

Thanks

[scholzj]

The connector log is part of the Connect log. What does it log or doesn't log depends to large extent on the connector itself.

[smeeklai]

So if a connector actually does the job, technically it supposes to logs something in the Connect log right? Not only this connector but the others as well.

[scholzj]

Well, the connector does the logging. It is up to its authors what kind of logging it does or whether they added any at all. Kafka Connect can give ou only general log messages about Connect. But if you want some messages such as 1000 records written to S3 bucker my-bucket, that is up to the connector. I'm afraid I have no idea what does Confluent S3SinkConnector log.

[smeeklai]

Thanks. I used this connector somewhere else before so I know it will log something if it's working. So If I assume that's the case, where am I expected to find those logs? In the connect pod itself?

[smeeklai]

I found out that the connector didn't run due to an incorrect config. Now it runs and I can see the logs in the Connect pod. Thanks for your help.