Failed to parse default container image configuration for Kafka from environment variable STRIMZI_KAFKA_IMAGES

[Muddam234]

Hi All

I am planning to migrate my Strimzi cluster to (0.23 to 0.26) and i have migrated my CRDs to 0.26 successful. and i am planning to migrate Kafka operator and i am getting an issue. Can some one please help me this issue.

PS C:\Users\MUPUL\Desktop\2.8 to 3.0> kubectl apply -f strimzi-crds-0.26.0.yaml
W1110 18:51:39.940821 17388 gcp.go:120] WARNING: the gcp auth plugin is deprecated in v1.22+, unavailable in v1.25+; use gcloud instead.
To learn more, consult https://cloud.google.com/blog/products/containers-kubernetes/kubectl-auth-changes-in-gke
customresourcedefinition.apiextensions.k8s.io/kafkas.kafka.strimzi.io configured
customresourcedefinition.apiextensions.k8s.io/kafkaconnects.kafka.strimzi.io configured
customresourcedefinition.apiextensions.k8s.io/kafkatopics.kafka.strimzi.io unchanged
customresourcedefinition.apiextensions.k8s.io/kafkausers.kafka.strimzi.io configured
customresourcedefinition.apiextensions.k8s.io/kafkamirrormakers.kafka.strimzi.io configured
customresourcedefinition.apiextensions.k8s.io/kafkabridges.kafka.strimzi.io configured
customresourcedefinition.apiextensions.k8s.io/kafkaconnectors.kafka.strimzi.io unchanged
customresourcedefinition.apiextensions.k8s.io/kafkamirrormaker2s.kafka.strimzi.io configured
customresourcedefinition.apiextensions.k8s.io/kafkarebalances.kafka.strimzi.io unchanged
PS C:\Users\MUPUL\Desktop\2.8 to 3.0>

helm upgrade --install strimzi-kafka-operator strimzi/strimzi-kafka-operator --namespace strimzi --version 0.26.0

LOGS:

Auto-detected KUBERNETES_SERVICE_DNS_DOMAIN: cluster.local

• export MALLOC_ARENA_MAX=2
• MALLOC_ARENA_MAX=2
• JAVA_OPTS=' -Dlog4j2.configurationFile=file:/opt/strimzi/custom-config/log4j2.properties -Dvertx.cacheDirBase=/tmp/vertx-cache -Djava.security.egd=file:/dev/./urandom'
  ++ get_gc_opts
  ++ '[' '' == true ']'
  ++ echo ''
• JAVA_OPTS=' -Dlog4j2.configurationFile=file:/opt/strimzi/custom-config/log4j2.properties -Dvertx.cacheDirBase=/tmp/vertx-cache -Djava.security.egd=file:/dev/./urandom '
• JAVA_OPTS=' -Dlog4j2.configurationFile=file:/opt/strimzi/custom-config/log4j2.properties -Dvertx.cacheDirBase=/tmp/vertx-cache -Djava.security.egd=file:/dev/./urandom --illegal-access=deny'
• exec /usr/bin/tini -w -e 143 -- java -Dlog4j2.configurationFile=file:/opt/strimzi/custom-config/log4j2.properties -Dvertx.cacheDirBase=/tmp/vertx-cache -Djava.security.egd=file:/dev/./urandom --illegal-access=deny -classpath lib/io.strimzi.cluster-operator-0.26.0.jar:lib/io.prometheus.simpleclient_common-0.7.0.jar:lib/io.fabric8.kubernetes-model-policy-5.8.0.jar:lib/io.strimzi.api-0.26.0.jar:lib/io.fabric8.kubernetes-model-batch-5.8.0.jar:lib/io.fabric8.openshift-model-storageversionmigrator-5.8.0.jar:lib/io.fabric8.openshift-model-machineconfig-5.8.0.jar:lib/io.fabric8.kubernetes-model-certificates-5.8.0.jar:lib/io.strimzi.config-model-0.26.0.jar:lib/io.fabric8.kubernetes-model-node-5.8.0.jar:lib/io.netty.netty-handler-proxy-4.1.68.Final.jar:lib/io.netty.netty-transport-native-epoll-4.1.68.Final.jar:lib/com.squareup.okio.okio-1.15.0.jar:lib/io.netty.netty-codec-socks-4.1.68.Final.jar:lib/io.netty.netty-codec-dns-4.1.68.Final.jar:lib/io.netty.netty-transport-4.1.68.Final.jar:lib/io.fabric8.openshift-model-miscellaneous-5.8.0.jar:lib/org.apache.kafka.kafka-clients-3.0.0.jar:lib/org.hdrhistogram.HdrHistogram-2.1.11.jar:lib/com.fasterxml.jackson.core.jackson-databind-2.11.3.jar:lib/io.netty.netty-transport-native-epoll-4.1.68.Final-linux-x86_64.jar:lib/io.netty.netty-transport-native-unix-common-4.1.68.Final.jar:lib/io.fabric8.kubernetes-model-metrics-5.8.0.jar:lib/org.apache.logging.log4j.log4j-core-2.14.1.jar:lib/io.prometheus.simpleclient-0.7.0.jar:lib/io.netty.netty-resolver-dns-4.1.68.Final.jar:lib/io.strimzi.operator-common-0.26.0.jar:lib/com.squareup.okhttp3.logging-interceptor-3.12.12.jar:lib/org.apache.yetus.audience-annotations-0.5.0.jar:lib/io.fabric8.kubernetes-model-scheduling-5.8.0.jar:lib/com.fasterxml.jackson.core.jackson-annotations-2.11.3.jar:lib/io.fabric8.kubernetes-model-admissionregistration-5.8.0.jar:lib/org.apache.zookeeper.zookeeper-jute-3.6.3.jar:lib/io.fabric8.openshift-model-operatorhub-5.8.0.jar:lib/io.fabric8.kubernetes-model-apps-5.8.0.jar:lib/io.micrometer.micrometer-core-1.3.1.jar:lib/io.fabric8.openshift-model-console-5.8.0.jar:lib/io.fabric8.kubernetes-model-extensions-5.8.0.jar:lib/io.fabric8.kubernetes-model-apiextensions-5.8.0.jar:lib/io.netty.netty-resolver-4.1.68.Final.jar:lib/org.apache.zookeeper.zookeeper-3.6.3.jar:lib/io.strimzi.certificate-manager-0.26.0.jar:lib/io.fabric8.kubernetes-model-events-5.8.0.jar:lib/net.minidev.accessors-smart-2.4.7.jar:lib/io.fabric8.kubernetes-model-core-5.8.0.jar:lib/com.fasterxml.jackson.dataformat.jackson-dataformat-yaml-2.10.5.jar:lib/com.nimbusds.nimbus-jose-jwt-9.10.jar:lib/io.fabric8.openshift-model-clusterautoscaling-5.8.0.jar:lib/io.fabric8.kubernetes-model-discovery-5.8.0.jar:lib/io.vertx.vertx-micrometer-metrics-4.1.5.jar:lib/io.netty.netty-codec-4.1.68.Final.jar:lib/io.fabric8.kubernetes-model-flowcontrol-5.8.0.jar:lib/com.github.mifmif.generex-1.0.2.jar:lib/io.fabric8.openshift-model-5.8.0.jar:lib/org.xerial.snappy.snappy-java-1.1.8.1.jar:lib/io.fabric8.openshift-client-5.8.0.jar:lib/com.fasterxml.jackson.core.jackson-core-2.11.3.jar:lib/com.github.stephenc.jcip.jcip-annotations-1.0-1.jar:lib/io.fabric8.kubernetes-model-rbac-5.8.0.jar:lib/org.yaml.snakeyaml-1.26.jar:lib/io.netty.netty-handler-4.1.68.Final.jar:lib/io.strimzi.kafka-oauth-server-plain-0.8.1.jar:lib/io.strimzi.crd-annotations-0.26.0.jar:lib/io.fabric8.openshift-model-whereabouts-5.8.0.jar:lib/io.fabric8.kubernetes-model-coordination-5.8.0.jar:lib/org.lz4.lz4-java-1.7.1.jar:lib/io.fabric8.kubernetes-client-5.8.0.jar:lib/io.fabric8.kubernetes-model-autoscaling-5.8.0.jar:lib/io.vertx.vertx-core-4.1.5.jar:lib/dk.brics.automaton.automaton-1.11-8.jar:lib/io.netty.netty-codec-http-4.1.68.Final.jar:lib/io.strimzi.kafka-oauth-client-0.8.1.jar:lib/io.fabric8.openshift-model-operator-5.8.0.jar:lib/io.fabric8.kubernetes-model-common-5.8.0.jar:lib/io.fabric8.openshift-model-tuned-5.8.0.jar:lib/io.netty.netty-common-4.1.68.Final.jar:lib/com.jayway.jsonpath.json-path-2.6.0.jar:lib/net.minidev.json-smart-2.4.7.jar:lib/io.netty.netty-buffer-4.1.68.Final.jar:lib/io.fabric8.zjsonpatch-0.3.0.jar:lib/io.strimzi.kafka-oauth-server-0.8.1.jar:lib/io.strimzi.kafka-oauth-common-0.8.1.jar:lib/io.fabric8.openshift-model-monitoring-5.8.0.jar:lib/com.squareup.okhttp3.okhttp-3.12.6.jar:lib/io.fabric8.kubernetes-model-storageclass-5.8.0.jar:lib/io.fabric8.openshift-model-machine-5.8.0.jar:lib/org.apache.logging.log4j.log4j-slf4j-impl-2.14.1.jar:lib/org.apache.logging.log4j.log4j-api-2.14.1.jar:lib/org.slf4j.slf4j-api-1.7.25.jar:lib/org.latencyutils.LatencyUtils-2.0.3.jar:lib/io.netty.netty-codec-http2-4.1.68.Final.jar:lib/com.github.luben.zstd-jni-1.5.0-2.jar:lib/com.fasterxml.jackson.datatype.jackson-datatype-jsr310-2.11.2.jar:lib/io.fabric8.kubernetes-model-networking-5.8.0.jar:lib/io.micrometer.micrometer-registry-prometheus-1.3.1.jar:lib/org.quartz-scheduler.quartz-2.3.2.jar io.strimzi.operator.cluster.Main
  2023-11-10 19:25:17 INFO Main:60 - ClusterOperator 0.26.0 is starting
  Exception in thread "main" io.strimzi.operator.common.InvalidConfigurationException: Failed to parse default container image configuration for Kafka from environment variable STRIMZI_KAFKA_IMAGES
  at io.strimzi.operator.cluster.ClusterOperatorConfig.parseKafkaVersions(ClusterOperatorConfig.java:367)
  at io.strimzi.operator.cluster.ClusterOperatorConfig.fromMap(ClusterOperatorConfig.java:158)
  at io.strimzi.operator.cluster.Main.main(Main.java:61)
  Caused by: io.strimzi.operator.cluster.model.NoImageException: No image for version 2.8.0 in {3.5.0=quay.io/strimzi/kafka:0.38.0-kafka-3.5.0, 3.5.1=quay.io/strimzi/kafka:0.38.0-kafka-3.5.1, 3.6.0=quay.io/strimzi/kafka:0.38.0-kafka-3.6.0}
  at io.strimzi.operator.cluster.model.KafkaVersion$Lookup.image(KafkaVersion.java:164)
  at io.strimzi.operator.cluster.model.KafkaVersion$Lookup.validateImages(KafkaVersion.java:200)
  at io.strimzi.operator.cluster.model.KafkaVersion$Lookup.validateKafkaImages(KafkaVersion.java:217)
  at io.strimzi.operator.cluster.ClusterOperatorConfig.parseKafkaVersions(ClusterOperatorConfig.java:353)
  ... 2 more
  Stream closed EOF for strimzi/strimzi-cluster-operator-78c9bd8fcb-gdvjv (strimzi-cluster-operator)

[Muddam234]

@scholzj @katheris Can you please help me on this

[scholzj]

The error suggests that you try to use Strimzi 0.26 (as you said). But in the STRIMZI_KAFKA_IMAGES variable, you have the Kafka versions configured from Strimzi 0.38. That does not work - you need to use the Kafka versions shipped with Strimzi 0.26. Unfortunately, from what you shared, I have no idea why or how this happened => whether this is something you changed yourself or whether this is something what Helm screwed up.

[scholzj]

Just to add -> to fix this, you have to configure the 0.26 Deployment to use the 0.26 Kafka versions: https://github.com/strimzi/strimzi-kafka-operator/blob/0.26.0/install/cluster-operator/060-Deployment-strimzi-cluster-operator.yaml#L58

[Muddam234]

I did not changed anything i just run the below commands

helm repo add strimzi https://strimzi.io/charts/
helm upgrade --install strimzi-kafka-operator strimzi/strimzi-kafka-operator --namespace strimzi --version 0.26.0

and the same command works for 0.23.0
helm upgrade --install strimzi-kafka-operator strimzi/strimzi-kafka-operator --namespace strimzi --version 0.23.0

[scholzj]

I guess that Helm does something wrong then. I do not use Helm but the YAML files, so I cannot help much more. Sorry.

[Muddam234]

Ok thanks @scholzj let me try to figure out this