Creating a KafkaConnect image from a maven repository

[anamaymane]

I'm trying too build a connect image using maven for retrieving an elasticsearch-connector. But apparently strimzi doesn't recognize the used configuration:

Here is my KafkaConnect yaml file:

apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaConnect
metadata:
  name: my-kafka-cluster-connect
  annotations:
    strimzi.io/use-connector-resources: "true"
spec:
  replicas: 2
  bootstrapServers: my-cluster-kafka-bootstrap:9092
  build:
    output:
      type: docker
      image: registry.gitlab.tech/test/kafka-connect-strimzi:1.6
      pushSecret: regcred
    plugins:
      - name: kafka-connect-elasticsearch2
        artifacts:
          - type: maven
            groupId: io.confluent
            artifactId: kafka-connect-elasticsearch
            version: 10.0.1

  config:
      group.id: connect-kafka-to-elk
      config.storage.topic: docker-connect-configs
      offset.storage.topic: docker-connect-offsets
      status.storage.topic: docker-connect-status
      key.ignore: "true"
      schema.ignore: "true"
      value.converter: "org.apache.kafka.connect.json.JsonConverter"
      value.converter.schemas.enable: false
      config.storage.replication.factor: "1"
      offset.storage.replication.factor: "1"
      status.storage.replication.factor: "1"
  resources:
    requests:
      cpu: "2"
      memory: 4Gi
    limits:
      cpu: "2"
      memory: 4Gi
  externalConfiguration:
    volumes:
      - name: elk-secret-store
        secret:
          secretName: elk-secret-store
  template:
    pod:
      imagePullSecrets:
        - name: regcred

Noting that I'm trying to use maven as a plugin source, because the connectors jars are missing some java dependencies

[scholzj]

What does it mean doesn't recognize the used configuration. I think you need to share more details, logs, statuses etc.

[anamaymane]

When I try to create a jar from a maven repo like this:

build:
    output:
      type: docker
      image: registry.gitlab.tech/test/kafka-connect-strimzi:1.6
      pushSecret: regcred
    plugins:
      - name: kafka-connect-elasticsearch
        artifacts:
          - type: maven
            groupId: io.confluent
            artifactId: kafka-connect-elasticsearch
            version: "10.0.1"

I have the following error when trying to update the cluster:

Error: UPGRADE FAILED: error validating "": error validating data: [ValidationError(KafkaConnect.spec.build.plugins[0].artifacts[0]): unknown field "artifactId" in io.strimzi.kafka.v1beta2.KafkaConnect.spec.build.plugins.artifacts, ValidationError(KafkaConnect.spec.build.plugins[0].artifacts[0]): unknown field "groupId" in io.strimzi.kafka.v1beta2.KafkaConnect.spec.build.plugins.artifacts]

I'm using strimzi 0.29.0

[anamaymane]

The problem is that the image is not built yet. the error occurs during the build

[anamaymane]

otherwise, is there a possiblity to pass env variables to the connect image builder pod. I think that specifying a maven option through environment variables will help me resolve the issue

[scholzj]

That doesn't matter. The Dockerfile is not the image ... that is the file with the instructions used to build the Docker image.

[anamaymane]

I finally find it, here is its content:

##############################
##############################
# This file is automatically generated by the Strimzi Cluster Operator
# Any changes to this file will be ignored and overwritten!
##############################
##############################

FROM quay.io/strimzi/maven-builder:0.29.0 AS downloadArtifacts
RUN 'curl' '-f' '-L' '--create-dirs' '--output' '/tmp/kafka-connect-elasticsearch/2cfc6ef7/pom.xml' 'https://packages.confluent.io/maven/io/confluent/kafka-connect-elasticsearch/10.0.1/kafka-connect-elasticsearch-10.0.1.pom' \
      && 'mvn' 'dependency:copy-dependencies' '-DoutputDirectory=/tmp/artifacts/kafka-connect-elasticsearch/2cfc6ef7' '-f' '/tmp/kafka-connect-elasticsearch/2cfc6ef7/pom.xml' \
      && 'curl' '-f' '-L' '--create-dirs' '--output' '/tmp/artifacts/kafka-connect-elasticsearch/2cfc6ef7/kafka-connect-elasticsearch-10.0.1.jar' 'https://packages.confluent.io/maven/io/confluent/kafka-connect-elasticsearch/10.0.1/kafka-connect-elasticsearch-10.0.1.jar'

FROM quay.io/strimzi/kafka:0.29.0-kafka-3.2.0

USER root:root

##########
# Connector plugin kafka-connect-elasticsearch
##########
COPY --from=downloadArtifacts '/tmp/artifacts/kafka-connect-elasticsearch/2cfc6ef7' '/opt/kafka/plugins/kafka-connect-elasticsearch/2cfc6ef7'

[scholzj]

Hmm, the pom.xml does not have the repository specified, so that is why it is not found. Not sure there is any easy workaround or fix.

[pcollinscode]

I ran across this same issue yesterday with the Confluent repository, here is what I did to solve it for the moment

1. I created a new Dockerfile which extended the Maven builder image to add a user's settings.xml file with additional repositories that Maven should use to look for dependencies.

FROM quay.io/strimzi/maven-builder

# I needed to use the root user, when I tried to use the 1001 user defined in the base it could not find the settings file. I didn't investigate further, see here https://github.com/strimzi/strimzi-kafka-operator/blob/main/docker-images/maven-builder/Dockerfile.
USER root

# see https://maven.apache.org/guides/mini/guide-multiple-repositories.html#repository-order for how Maven determines repos
COPY settings.xml /root/.m2/settings.xml

And then the settings.xml file that is copied into the users maven settings location

<settings xmlns="http://maven.apache.org/SETTINGS/1.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
      xsi:schemaLocation="http://maven.apache.org/SETTINGS/1.0.0 https://maven.apache.org/xsd/settings-1.0.0.xsd">
    <profiles>
        <profile>
            <id>confluent</id>
            <repositories>
                <repository>
                    <id>confluent</id>
                    <name>Confluent</name>
                    <url>https://packages.confluent.io/maven</url>
                </repository>
            </repositories>
        </profile>
    </profiles>
    <activeProfiles>
        <activeProfile>confluent</activeProfile>
    </activeProfiles>
</settings>

2. Update the Strimzi operator use the extended Maven builder (swap out whichever image registry you are using). NOTE: Your k8s cluster needs to be configured to pull images from your repository if its private first.

resource "helm_release" "strimzi_operators" {
  name       = "strimzi-operator"
  repository = "https://strimzi.io/charts/"
  chart      = "strimzi-kafka-operator"
  version    = "0.28.0"
  namespace  = local.ns_name
  
  set {
    name  = "mavenBuilder.image.registry"
    value = "<your registry here>"
  }
  set {
    name  = "mavenBuilder.image.repository"
    value = "<your repo here>"
  }
  set {
    name  = "mavenBuilder.image.name"
    value = "maven-builder"
  }
  set {
    name  = "mavenBuilder.image.tag"
    value = "latest"
  }
}

3. Added the maven artifact to the KafkaConnect plugins

    plugins:
      - name: debezium-postgres-connector
        artifacts:
          - type: tgz
            url: https://repo1.maven.org/maven2/io/debezium/debezium-connector-postgres/{{ .Values.debezium.version }}.Final/debezium-connector-postgres-{{ .Values.debezium.version }}.Final-plugin.tar.gz
      - name: confluent-connect-avro-converter
        artifacts:
          - type: maven
            repository: https://packages.confluent.io/maven
            group: io.confluent
            artifact: kafka-connect-avro-converter
            version: 7.1.1

And that was it and it successfully built and deployed KafkaConnect with Confluent plugins from their registry even though their pom.xml don't specify the registry which is what causes the issue originally.

It could be nice if Strimzi somehow allowed us to specify the maven-builder user settings via the chart and then build that settings.xml file within the default maven-builder image when people might need to configure Maven for any reason.

[jcrompton-avetta]

Your error message told you what the problem was:

unknown field "artifactId"

Change

groupId: io.confluent
artifactId: kafka-connect-elasticsearch

to (remove "Id")

group: io.confluent
artifact: kafka-connect-elasticsearch