Kafka Connectors Artifact locations

[twosdai]

Hey I'm new to Kafka and Kafka connect. Specifically I'm looking for Influxdb, and AWS kinesis connectors.
I know that in ConfluentHub they have them available
https://www.confluent.io/hub/confluentinc/kafka-connect-influxdb
https://www.confluent.io/hub/confluentinc/kafka-connect-kinesis

However I don't see an easy URL link that I can paste into the build properties of my KafkaConnect configuration when deploying my cluster
similar to the documentation:

plugins: # (3)
      - name: debezium-postgres-connector
        artifacts:
          - type: tgz
            url: https://repo1.maven.org/maven2/io/debezium/debezium-connector-postgres/1.3.1.Final/debezium-connector-postgres-1.3.1.Final-plugin.tar.gz
            sha512sum: 962a12151bdf9a5a30627eebac739955a4fd95a08d373b86bdcea2b4d0c27dd6e1edd5cb548045e115e33a9e69b1b2a352bee24df035a0447cb820077af00c03
      - name: camel-telegram
        artifacts:
          - type: tgz
            url: https://repo.maven.apache.org/maven2/org/apache/camel/kafkaconnector/camel-telegram-kafka-connector/0.7.0/camel-telegram-kafka-connector-0.7.0-package.tar.gz
            sha512sum: a9b1ac63e3284bea7836d7d24d84208c49cdf5600070e6bd1535de654f6920b74ad950d51733e8020bf4187870699819f54ef5859c7846ee4081507f48873479

I was thinking of just creating my own image, where I install the connectors I need and then tell the strimzi operator to just use that, however I just wanted to check if there was something obvious I was missing.

Any thoughts or help would be greatly appreciated, also let me know if I'm completely off base with my thinking. Very new to this ecosystem.

[scholzj]

You can of course build your own image instead of using the Kafka Connect Build as well -> check the docs which have a guide. But for the download links, I guess you would need to ask the connector authors.

[twosdai]

Ahh ok I did a bit more research.

So for connectors that are associated with confluent-hub I am assuming that strimzi isn't able to specify to pull directly from confluent-hub similar to what is offered with the confluent operator.

for example something like the syntax confluent uses here:
https://github.com/confluentinc/confluent-kubernetes-examples/blob/master/connector/datagen-source-connector/confluent-platform.yaml#L42-L50

Do you know if its still possible to build the image with connectors listed in confluent hub into the connect image? Or are these incompatible generally.

Sorry for the potentially obvious questions, I'm trying to figure out what the lines are within this whole ecosystem and its not always clear what is basically an enterprise feature provided by confluent and what is available as FOSS.

[scholzj]

Confluent Hub is a proprietary Confluent thing. It does not have clear licensing or compatibility with Apache Kafka. And last time I checked, the website code did obfuscation of the download URLs which to me suggests they do not want it to be used by others (in any case, using such URLs with some decoding from the website would not provide the required stability / reliability). If I'm wrong and they provide some clear licensing and a stable API, we can look into it and add support for Confluent Hub.

As described in the Strimzi docs, you can add artifacts as URLs to JARs, ZIP files or TAR.GZ files or as Maven coordinates to the Maven Central repository.

[ee07dazn]

build ur own custom image and use confluent hub
see sample below

FROM confluentinc/cp-kafka-connect:7.1.4 as hub
RUN confluent-hub install --no-prompt wepay/kafka-connect-bigquery:2.4.0
RUN confluent-hub install --no-prompt confluentinc/kafka-connect-avro-converter:7.2.1
FROM quay.io/strimzi/kafka:0.30.0-kafka-3.2.0
USER root:root
RUN mkdir -p /opt/kafka/plugins/bq-sink-connector && mkdir -p /opt/kafka/plugins/avro && mkdir -p /opt/kafka/plugins/gcs-sink-connector
COPY --from=hub /usr/share/confluent-hub-components/wepay-kafka-connect-bigquery/lib /opt/kafka/plugins/bq-sink-connector/
COPY --from=hub /usr/share/confluent-hub-components/confluentinc-kafka-connect-avro-converter/lib /opt/kafka/plugins/avro/
COPY ./connectors/aiven-kafka-connect-gcs-0.9.0.tar.gz /opt/kafka/plugins/gcs-sink-connector/
RUN tar -xvzf /opt/kafka/plugins/gcs-sink-connector/aiven-kafka-connect-gcs-0.9.0.tar.gz --directory /opt/kafka/plugins/gcs-sink-connector
RUN rm /opt/kafka/plugins/gcs-sink-connector/aiven-kafka-connect-gcs-0.9.0.tar.gz
USER 1001

[ee07dazn]

also be aware of confluence license model ...the one's i have above are Free for the community, but there are a bunch that needs licensing.