KafkaConnect Kaniko build fails with filesystem permission error

[andreas-ibm]

Hello, I'm still trying to get my head around KafkaConnect stuffs, so apologies if I've made incorrect assumptions.

My goal: Run Debezium CDC against Oracle database.

I'll jump straight to the error to scintillate the readers, then pedal back to explain how I got there:

INFO[0000] Retrieving image manifest quay.io/strimzi/kafka:0.29.0-kafka-3.1.0
INFO[0000] Retrieving image quay.io/strimzi/kafka:0.29.0-kafka-3.1.0 from registry quay.io
INFO[0001] Built cross stage deps: map[]
INFO[0001] Retrieving image manifest quay.io/strimzi/kafka:0.29.0-kafka-3.1.0
INFO[0001] Returning cached image manifest
INFO[0001] Executing 0 build triggers
INFO[0001] Unpacking rootfs as cmd RUN 'mkdir' '-p' '/opt/kafka/plugins/debezium-oracle-connector/3bf764c2'       && 'curl' '-f' '-L' '--output' '/opt/kafka/plugins/debezium-oracle-connector/3bf764c2.tgz' 'https://repo1.maven.org/maven2/io/debezium/debezium-connector-oracle/1.9.5.Final/debezium-connector-oracle-1.9.5.Final-plugin.tar.gz'       && 'tar' 'xvfz' '/opt/kafka/plugins/debezium-oracle-connector/3bf764c2.tgz' '-C' '/opt/kafka/plugins/debezium-oracle-connector/3bf764c2'       && 'rm' '-vf' '/opt/kafka/plugins/debezium-oracle-connector/3bf764c2.tgz' requires it.
error building image: error building stage: failed to get filesystem from image: failed to write "security.capability" attribute to "/usr/bin/newgidmap": operation not permitted

Stack:

• 4x Ubuntu 20.04.4 LTS nodes; 3 of which are worker nodes
• CRI-O 1.23
• Kubernetes 1.23
• Strimzi 0.29.0
• Ceph

Kafka Environment:

NAME                                                   READY   STATUS    RESTARTS       AGE
pod/debezium-connect-cluster-connect-95db54f5f-c9kmb   1/1     Running   0              2d17h
pod/kafka-consumer                                     1/1     Running   0              43h
pod/kafka-consumer-2                                   1/1     Running   0              43h
pod/my-cluster-entity-operator-77bcc4b67f-szwfv        3/3     Running   3 (2d7h ago)   2d7h
pod/my-cluster-kafka-0                                 1/1     Running   0              2d7h
pod/my-cluster-zookeeper-0                             1/1     Running   2 (2d7h ago)   2d7h
pod/strimzi-cluster-operator-86864b86d5-g6964          1/1     Running   0              2d7h
pod/watcher                                            1/1     Running   0              43h

NAME                                           TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)                               AGE
service/debezium-connect-cluster-connect-api   ClusterIP   10.99.114.193   <none>        8083/TCP                              2d17h
service/my-cluster-kafka-bootstrap             ClusterIP   10.109.24.240   <none>        9091/TCP,9092/TCP,9093/TCP            79d
service/my-cluster-kafka-brokers               ClusterIP   None            <none>        9090/TCP,9091/TCP,9092/TCP,9093/TCP   79d
service/my-cluster-zookeeper-client            ClusterIP   10.98.240.146   <none>        2181/TCP                              79d
service/my-cluster-zookeeper-nodes             ClusterIP   None            <none>        2181/TCP,2888/TCP,3888/TCP            79d

NAME                                               READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/debezium-connect-cluster-connect   1/1     1            1           2d17h
deployment.apps/my-cluster-entity-operator         1/1     1            1           79d
deployment.apps/strimzi-cluster-operator           1/1     1            1           79d

NAME                                                         DESIRED   CURRENT   READY   AGE
replicaset.apps/debezium-connect-cluster-connect-95db54f5f   1         1         1       2d17h
replicaset.apps/my-cluster-entity-operator-77bcc4b67f        1         1         1       79d
replicaset.apps/strimzi-cluster-operator-86864b86d5          1         1         1       79d

NAME                                    READY   AGE
statefulset.apps/my-cluster-kafka       1/1     79d
statefulset.apps/my-cluster-zookeeper   1/1     79d

i.e. I have a simple cluster with the strimzi operator, a debezium KafkaConnect cluster (using a pre-built mysql image) and some basic consumers and producers. I have MySQL, DB2 and Oracle running in other namespaces.

I deploy the following KafkaConnect yaml:

apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaConnect
metadata:
  name: debezium-oracle-cluster
  namespace: kafka
  annotations:
    strimzi.io/use-connector-resources: "true"
spec:
  version: 3.1.0
  replicas: 1
  bootstrapServers: my-cluster-kafka-bootstrap:9092
  config:
    config.providers: secrets
    config.providers.secrets.class: io.strimzi.kafka.KubernetesSecretConfigProvider
    group.id: connect-cluster
    offset.storage.topic: connect-cluster-offsets
    config.storage.topic: connect-cluster-configs
    status.storage.topic: connect-cluster-status
    # -1 means it will use the default replication factor configured in the broker
    config.storage.replication.factor: -1
    offset.storage.replication.factor: -1
    status.storage.replication.factor: -1
  build:
    output:
      type: docker
      image: uk.icr.io/debezium/debezium-connect-oracle:latest
      #additionalKanikoOptions: [--insecure]
      pushSecret: ibmcloud-credentials
    plugins:
      - name: debezium-oracle-connector
        artifacts:
          - type: tgz
            url: https://repo1.maven.org/maven2/io/debezium/debezium-connector-oracle/1.9.5.Final/debezium-connector-oracle-1.9.5.Final-plugin.tar.gz
  externalConfiguration:
    volumes:
      - name: connector-config
        secret:
          secretName: oracle-credentials

I might have done things wrong here, I've tried lots of things (I have already tried this when creating a DB2 connector).

Once I apply the yaml I get a debezium-oracle-cluster-connect-build pod and the log at the beginning is from that.

So I suspect it's not expecting to build in a CRI-O environment maybe? Not sure what else I can do to debug this (I'm unfamiliar with Kaniko too).
Ultimately, this Issue might belong with Kaniko or Debezium (or plain User Error), feel free to tell me to go someplace else :-) I don't know whether the attempt to set the security.capability property on the file is a base Kaniko thing or some layer on top.

[scholzj]

I personally haven't done any testing on CRI-O. My servers normally use Containerd. But I'm not aware of any limitations. From the error, it sounds to me like someone / something restricted the permissions of the Kaniko container. So maybe you should check if you don't have any tools injecting some restricted security context etc. These might not work well with Kaniko since currently it requires some capabilities and the ability to write to some directories of the image etc.

If needed, keep in mind that you can also use the traditional Dockerfile approach and simply build the container image manually or in some CI. That is something what is independent on Kaniko and should always work.

[andreas-ibm]

Thanks, the "traditional" approach (well, using podman, not docker) got me going with DB2, but I thought I'd try to figure it out properly when it happened again :-)
I've seen an Issue in the Kaniko git for when they started using security.capability, so i think I'll head over there and see if they can help me.
A quick question if I may on how the KafkaConnect image is built up: Can I find the "base" image that the plugins are extracted on top of? Is that the quay.io/strimzi/kafka:0.29.0-kafka-3.1.0 image?

[andreas-ibm]

And another question; can you think of a way I could reproduce this outside Strimzi? How can I see the Dockerfile it's trying to use?
Is it essentially:

FROM quay.io/strimzi/kafka:0.29.0-kafka-3.1.0

USER root:root

RUN 'mkdir' '-p' '/opt/kafka/plugins/debezium-oracle-connector/3bf764c2'       && 'curl' '-f' '-L' '--output' '/opt/kafka/plugins/debezium-oracle-connector/3bf764c2.tgz' 'https://repo1.maven.org/maven2/io/debezium/debezium-connector-oracle/1.9.5.Final/debezium-connector-oracle-1.9.5.Final-plugin.tar.gz'       && 'tar' 'xvfz' '/opt/kafka/plugins/debezium-oracle-connector/3bf764c2.tgz' '-C' '/opt/kafka/plugins/debezium-oracle-connector/3bf764c2'       && 'rm' '-vf' '/opt/kafka/plugins/debezium-oracle-connector/3bf764c2.tgz' 

?

[scholzj]

Well, when you create the KafkaConnect CR with the build section, it will generate the Dockerfile. You can find it in a ConfigMap. It will be named <KafkaConnectName>-connect-dockerfile. But there is no way to get the Dockerfile without creating the KafkaConnect CR first.

[andreas-ibm]

got it, thanks:

    ##############################
    ##############################
    # This file is automatically generated by the Strimzi Cluster Operator
    # Any changes to this file will be ignored and overwritten!
    ##############################
    ##############################

    FROM quay.io/strimzi/kafka:0.29.0-kafka-3.1.0

    USER root:root

    ##########
    # Connector plugin debezium-oracle-connector
    ##########
    RUN 'mkdir' '-p' '/opt/kafka/plugins/debezium-oracle-connector/3bf764c2' \
          && 'curl' '-f' '-L' '--output' '/opt/kafka/plugins/debezium-oracle-connector/3bf764c2.tgz' 'https://repo1.maven.org/maven2/io/debezium/debezium-connector-oracle/1.9.5.Final/debezium-connector-oracle-1.9.5.Final-plugin.tar.gz' \
          && 'tar' 'xvfz' '/opt/kafka/plugins/debezium-oracle-connector/3bf764c2.tgz' '-C' '/opt/kafka/plugins/debezium-oracle-connector/3bf764c2' \
          && 'rm' '-vf' '/opt/kafka/plugins/debezium-oracle-connector/3bf764c2.tgz'

    USER 1001

[andreas-ibm]

Kaniko issue here: GoogleContainerTools/kaniko#2201

[andreas-ibm]

I was given a workaround from the Kaniko issue:
I needed to add the following to the Pod definition:

    securityContext:
      capabilities:
        add: ["CHOWN", "DAC_OVERRIDE","FOWNER","SETFCAP","SETGID","SETUID"]

(I'll try to run with one at a time to see if there's a minimal set I need)

Is there a way to add those to the KafkaConnect yaml?

[scholzj]

You can configure the security contexts in the template sections.

[andreas-ibm]

thanks,
For the record, for anyone coming here from a search engine. This was the resulting yaml that worked:

apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaConnect
metadata:
  name: debezium-oracle-cluster
  namespace: kafka
  annotations:
    strimzi.io/use-connector-resources: "true"
spec:
  version: 3.1.0
  replicas: 1
  bootstrapServers: my-cluster-kafka-bootstrap:9092
  config:
    config.providers: secrets
    config.providers.secrets.class: io.strimzi.kafka.KubernetesSecretConfigProvider
    group.id: connect-cluster
    offset.storage.topic: connect-cluster-offsets
    config.storage.topic: connect-cluster-configs
    status.storage.topic: connect-cluster-status
    # -1 means it will use the default replication factor configured in the broker
    config.storage.replication.factor: -1
    offset.storage.replication.factor: -1
    status.storage.replication.factor: -1
  build:
    output:
      type: docker
      image: local.artifactory.devops/test/debezium-connect-oracle:latest
      pushSecret: jfrogcred
    plugins:
      - name: debezium-oracle-connector
        artifacts:
          - type: tgz
            url: https://repo1.maven.org/maven2/io/debezium/debezium-connector-oracle/1.9.5.Final/debezium-connector-oracle-1.9.5.Final-plugin.tar.gz
  template:
    pod:
      imagePullSecrets:
        - name: jfrogcred
    buildContainer:
      securityContext:
        capabilities:
          add: ["CHOWN", "DAC_OVERRIDE","FOWNER","SETFCAP","SETGID","SETUID"]
  externalConfiguration:
    volumes:
      - name: connector-config
        secret:
          secretName: oracle-credentials

In this I have config to:

• Create a cluster with Oracle drivers
• Push it to local artifactory virtual docker registry (private repo, hence needing both push and pull secrets)
• set security context to allow kaniko to build it.

hopefully this can help others!