[Question] Unable to pull kafka connect image from Azure container registry

[mukkchir]

Hello,

I am encountering a problem with pulling already built and pushed images of Kafka connect from the Azure container registry.
Below are the versions I'm using:

Strimzi: 0.34.0
Kafka: 3.4.0
Azure kubernetes: 1.26.3

Below is my dockerfile:

FROM quay.io/strimzi/kafka:0.34.0-kafka-3.4.0
USER root:root
COPY connectors /opt/kafka/plugins/
USER 1001

I have already created the pull-secret with the name public-acr using:

kubectl create secret docker-registry public-acr --namespace kafka --docker-server=REGISTRYNAME.azurecr.io --docker-username=NAME --docker-password=PASSWORD

I have followed these steps to setup strimzi:

1. kubectl create namespace kafka
2. kubectl create -f 'https://strimzi.io/install/latest?namespace=kafka' -n kafka
3. kubectl create -f .\kafka-persistent.yaml -n kafka
4. kubectl create -f .\kafka-connect.yaml -n kafka

kafka-persistent.yaml:

apiVersion: kafka.strimzi.io/v1beta2
kind: Kafka
metadata:
  name: CLUSTERNAME
spec:
  kafka:
    version: 3.4.0
    replicas: 3
    listeners:
      - name: plain
        port: 9092
        type: internal
        tls: false
      - name: tls
        port: 9093
        type: internal
        tls: true
      - name: external
        port: 9094
        type: loadbalancer
        tls: true
    config:
      offsets.topic.replication.factor: 1
      transaction.state.log.replication.factor: 1
      transaction.state.log.min.isr: 1
      default.replication.factor: 1
      min.insync.replicas: 1
      inter.broker.protocol.version: "3.4"
    storage:
      type: jbod
      volumes:
      - id: 0
        type: persistent-claim
        size: 10Gi
        deleteClaim: false
  zookeeper:
    replicas: 3
    storage:
      type: persistent-claim
      size: 10Gi
      deleteClaim: false
  entityOperator:
    topicOperator: {}
    userOperator: {}

The kafka connect yaml is where it gets interesting:

apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaConnect
metadata:
  name: kafka-acr-cluster
  annotations:
    strimzi.io/use-connector-resources: "true"
spec:
  image: REGISTRYNAME.azurecr.io/IMAGE_NAME:VERSION
  version: 3.4.0
  replicas: 1
  bootstrapServers: BOOTSTRAPNAME-kafka-bootstrap:9092
  logging:
    type: inline
    loggers:
      connect.root.logger.level: "DEBUG"
  config:
    group.id: acr-cluster
    offset.storage.topic: strimzi-acr-cluster-offsets
    config.storage.topic: strimzi-acr-cluster-configs
    status.storage.topic: strimzi-acr-cluster-status
  template:
    pod:
      imagePullSecrets:
        - name: public-acr

I used several different formats of using pull-secrets in the above kafka connect but just ended up in validation errors. I'm not sure if this schema is valid with template but there were no validation errors when deploying.

Yet I get the error after deploying kafka connect to the AKS Back-off pulling image and the pod remains in pending state.

I'm facing this problem only under KafkaConnect deployment type. Other deployments with ACR is already working.
Is my kafkaconnect yaml wrong?

[scholzj]

I think the custom resource looks good to me. What Strimzi does is that it basically just passes it on into the Pod configuration. As you did not shared the log, it is not clear if this is the only issue or not, but you can check if the Pod has the secret configured. And if it does, it means the secret has a wrong format or does not actually have the rights to pull the image. If the Pod does not have the secret configured, you should share the full cluster operator log to better understand why.

[mukkchir]

@scholzj Thanks for the reply. I have attached the cluster log here - strimzi-cluster-operator.log