Create Strimzi Kafka Connect image with Confluent kafka-connect-s3 plugin?

[kurtostfeld]

I'd like to use Kafka Connect to write data to Amazon S3, so I'd like to use the Confluent kafka-connect-s3-plugin with my Strimzi Kafka Cluster.

Is there a recommended way to do this?

I wrote a simple Dockerfile that downloads https://d1i4a15mxbxib1.cloudfront.net/api/plugins/confluentinc/kafka-connect-s3/versions/10.0.7/confluentinc-kafka-connect-s3-10.0.7.zip and unpacks into /opt/kafka/plugins/.

Then I found I need to get a separate jar file, kafka-connect-avro-converter-7.1.1.jar from the confluentinc-kafka-connect-avro plugin into the confluentinc-kafka-connect-s3-10.0.7/lib directory and everything seems to work. If I don't do this step, I get Class io.confluent.connect.avro.AvroConverter could not be found. exceptions at runtime.

What I have seems to work, but it seems like a hack, and I wanted to ask if there was simpler supported way to use the Confluent S3 plugin with Strimzi Kafka Connect?

[scholzj]

I don't think there is a simpler way to do this. You can try to use a different connector (e.g. the Apache Camel connector for S3). But it will not be 100% the same in terms of features or configuration.

I also know that some users use multistage Docker build to pull the connector with the Confluent command line tool in the first stage and then copy the connector to the next stage (see for example #3047 (comment)). But I never tried it my self.

[Rezkmike]

@kurtostfeld how you supply aws credentials/roles into the kafka connect? i have tried few methods by adding iam credentials and annotation eks.amazonaws.com/role

 template:
    pod:
      metadata:
        annotations:
          eks.amazonaws.com/role: "<ROLE ARN>"

    aws.access.key.id: 
    aws.secret.access.key:

but both methods are not works for me.

[Rezkmike]

able to make it work. eks.role seems not supported by confluent plugin.

but then its working using iam user access

aws.access.key.id: 
aws.secret.access.key:

p/s: please check your iam user policy/role and bucket policy, to make sure all the required permission is good.

add s3 log for troubleshoot

log4j.logger.io.confluent.connect.s3: INFO

[imrohankataria]

This is fairly easier, Use the Strimzi Kafka Connect build ferature.

I was in a similar situation where I wanted to add the Microsoft JDBC Jar into the Confluenct JDBC Plugin folder.
Ideally you want to install two jars in one folder, and this can be achieved by providing multiple artifacts for a plugin at docker image build level.

build:
    output:
      type: docker
      image: registry-link
      pushSecret: registry-secret
    plugins:
        - name: kafka-connect-jdbc
          artifacts:
            - type: jar
              url: https://packages.confluent.io/maven/io/confluent/kafka-connect-jdbc/10.7.0/kafka-connect-jdbc-10.7.0.jar
              version: 10.7.0
            - type: maven
              repository: https://repo1.maven.org/maven2/
              group: com.microsoft.sqlserver
              artifact: mssql-jdbc
              version: 12.2.0.jre11