"failed to get filesystem from image" when bootstraping a KafkaConnect instance

[itaydj]

Hi,
I am trying to follow Strimzi Documentation and launch a simple Kafka Connect instance using Strimzi.
I started by copying the basic example for here.

I adjusted it with my own configurations and ended up with this:

kind: KafkaConnect
metadata:
  name: itay-connect-cluster
  annotations:
    #    strimzi.io/use-connector-resources: "true"
spec:
  version: 2.8.0
  replicas: 1
  bootstrapServers: itay-kafka-kafka-bootstrap:9093
  tls:
    trustedCertificates:
      - secretName: itay-kafka-cluster-ca-cert
        certificate: ca.crt
  authentication:
    type: scram-sha-512
    username: connect-user
    passwordSecret:
      secretName: connect-user
      password: password
  build:
    output:
      type: docker
      image:docker.io/itaymiz/firstconnector:latest
      pushSecret: regcred
    plugins:
      - name: debezium-postgres-connector
        artifacts:
          - type: tgz
            url: https://repo1.maven.org/maven2/io/debezium/debezium-connector-postgres/1.3.1.Final/debezium-connector-postgres-1.3.1.Final-plugin.tar.gz
            sha512sum: 962a12151bdf9a5a30627eebac739955a4fd95a08d373b86bdcea2b4d0c27dd6e1edd5cb548045e115e33a9e69b1b2a352bee24df035a0447cb820077af00c03
  config:
    group.id: connect-cluster
    offset.storage.topic: connect-cluster-offsets
    config.storage.topic: connect-cluster-configs
    status.storage.topic: connect-cluster-status
    # -1 means it will use the default replication factor configured in the broker
    config.storage.replication.factor: -1
    offset.storage.replication.factor: -1
    status.storage.replication.factor: -1

When trying to apply this yaml to my cluster using kubectl apply, the pod tries to start but fail with:

E0812 09:35:54.764011       1 aws_credentials.go:77] while getting AWS credentials NoCredentialProviders: no valid providers in chain. Deprecated.
	For verbose messaging see aws.Config.CredentialsChainVerboseErrors
INFO[0007] Retrieving image manifest quay.io/strimzi/kafka:0.24.0-kafka-2.8.0 
INFO[0007] Retrieving image quay.io/strimzi/kafka:0.24.0-kafka-2.8.0 from registry quay.io 
INFO[0009] Built cross stage deps: map[]                
INFO[0009] Retrieving image manifest quay.io/strimzi/kafka:0.24.0-kafka-2.8.0 
INFO[0009] Returning cached image manifest              
INFO[0009] Executing 0 build triggers                   
INFO[0009] Unpacking rootfs as cmd RUN mkdir -p /opt/kafka/plugins/debezium-postgres-connector/99c2d3a0       && curl -L --output /opt/kafka/plugins/debezium-postgres-connector/99c2d3a0.tgz https://repo1.maven.org/maven2/io/debezium/debezium-connector-postgres/1.3.1.Final/debezium-connector-postgres-1.3.1.Final-plugin.tar.gz       && echo "962a12151bdf9a5a30627eebac739955a4fd95a08d373b86bdcea2b4d0c27dd6e1edd5cb548045e115e33a9e69b1b2a352bee24df035a0447cb820077af00c03 /opt/kafka/plugins/debezium-postgres-connector/99c2d3a0.tgz" > /opt/kafka/plugins/debezium-postgres-connector/99c2d3a0.tgz.sha512       && sha512sum --check /opt/kafka/plugins/debezium-postgres-connector/99c2d3a0.tgz.sha512       && rm -f /opt/kafka/plugins/debezium-postgres-connector/99c2d3a0.tgz.sha512       && tar xvfz /opt/kafka/plugins/debezium-postgres-connector/99c2d3a0.tgz -C /opt/kafka/plugins/debezium-postgres-connector/99c2d3a0       && rm -vf /opt/kafka/plugins/debezium-postgres-connector/99c2d3a0.tgz requires it. 
error building image: error building stage: failed to get filesystem from image: removing whiteout root/buildinfo/.wh..wh..opq: fstatat /root/buildinfo/.wh..opq: operation not permitted

Any tip will be much appreciated
Thanks

[scholzj]

I never saw this kind of error. Could it be that some weird security context is injected into the build pod or something?

[scholzj]

PS: You can always build the image manually from Dockerfile as a workaround.

[itaydj]

Thats actually running in our QA cluster where security is not enabled at all.

I know I can build the image by myself but I want to provide developers with an easy way of doing that, using the native Strimzi way.

Anyways, thanks alot, now that I know you didnt see such an error, I will continue investigating other paths.

Will update here when I find the issue.

[itaydj]

Just wanted to update that this issue is reproduced in all of our environments,
the only cluster I managed to execute this connect resource is my local kind cluster :-(
couldnt find much about this error online.
if anyone has any lead I would appreciate it alot.
Thanks