Operator not starting - "Back-off restarting failed container" #5241

sreejesh123 · 2021-07-01T17:11:42Z

sreejesh123
Jul 1, 2021

REMOVED IP ADDRESS from LOG

What happened:
deployment.apps/strimzi-cluster-operator created dkr-io/strimzi/operator:0.20.1
deployment.apps/strimzi-cluster-operator created
configmap/strimzi-cluster-operator created
serviceaccount/strimzi-cluster-operator created
clusterrole.rbac.authorization.k8s.io/strimzi-cluster-operator-namespaced created
clusterrole.rbac.authorization.k8s.io/strimzi-cluster-operator-global created
clusterrole.rbac.authorization.k8s.io/strimzi-kafka-broker created
clusterrole.rbac.authorization.k8s.io/strimzi-entity-operator created
clusterrole.rbac.authorization.k8s.io/strimzi-topic-operator created
clusterrole.rbac.authorization.k8s.io/strimzi-kafka-client created
clusterrolebinding.rbac.authorization.k8s.io/strimzi-cluster-operator created
clusterrolebinding.rbac.authorization.k8s.io/strimzi-cluster-operator-kafka-broker-delegation created
clusterrolebinding.rbac.authorization.k8s.io/strimzi-cluster-operator-kafka-client-delegation created
rolebinding.rbac.authorization.k8s.io/strimzi-cluster-operator created
rolebinding.rbac.authorization.k8s.io/strimzi-cluster-operator-entity-operator-delegation created
rolebinding.rbac.authorization.k8s.io/strimzi-cluster-operator-topic-operator-delegation created

I have all the CRD, Rolebinding, cluster role, customtrole , sa all created

What you expected to happen:
POD is suppose to start but it failed with "Back-off restarting failed container"

LOG from POD reads like this:

2021-07-01T16:32:03.2379573Z strimzi-cluster-operatorXXXXXX 0/1 CrashLoopBackOff 63 5h6m
2021-07-01T16:32:03.4950991Z + export MALLOC_ARENA_MAX=2
2021-07-01T16:32:03.4951645Z + MALLOC_ARENA_MAX=2
2021-07-01T16:32:03.4953908Z + JAVA_OPTS=' -Dlog4j2.configurationFile=file:/opt/strimzi/custom-config/log4j2.properties -Dvertx.cacheDirBase=/tmp -Djava.security.egd=file:/dev/./urandom'
2021-07-01T16:32:03.4954966Z ++ get_gc_opts
2021-07-01T16:32:03.4955500Z ++ '[' '' == true ']'
2021-07-01T16:32:03.4955982Z ++ echo ''
2021-07-01T16:32:03.4957284Z + JAVA_OPTS=' -Dlog4j2.configurationFile=file:/opt/strimzi/custom-config/log4j2.properties -Dvertx.cacheDirBase=/tmp -Djava.security.egd=file:/dev/./urandom '
2021-07-01T16:32:03.4958473Z ++ java -version
2021-07-01T16:32:03.4959144Z ++ sed -E -n 's/.* version "([0-9]).$/\1/p'
2021-07-01T16:32:03.4959599Z + JAVA_MAJOR_VERSION=11
2021-07-01T16:32:03.4960124Z + '[' 11 -ge 9 ']'
2021-07-01T16:32:03.4962178Z + JAVA_OPTS=' -Dlog4j2.configurationFile=file:/opt/strimzi/custom-config/log4j2.properties -Dvertx.cacheDirBase=/tmp -Djava.security.egd=file:/dev/./urandom --illegal-access=deny'
2021-07-01T16:32:03.5012322Z + exec /usr/bin/tini -w -e 143 -- java -Dlog4j2.configurationFile=file:/opt/strimzi/custom-config/log4j2.properties -Dvertx.cacheDirBase=/tmp -Djava.security.egd=file:/dev/./urandom --illegal-access=deny -classpath lib/io.strimzi.cluster-operator-0.20.1.jar:lib/com.fasterxml.jackson.core.jackson-core-2.10.4.jar:lib/io.prometheus.simpleclient_common-0.7.0.jar:lib/io.vertx.vertx-micrometer-metrics-3.9.1.jar:lib/io.strimzi.crd-annotations-0.20.1.jar:lib/io.fabric8.openshift-model-operator-4.12.0.jar:lib/org.apache.zookeeper.zookeeper-3.5.8.jar:lib/org.apache.logging.log4j.log4j-api-2.13.3.jar:lib/com.zaxxer.HikariCP-java7-2.4.13.jar:lib/org.keycloak.keycloak-core-11.0.2.jar:lib/io.fabric8.kubernetes-model-apps-4.12.0.jar:lib/org.bouncycastle.bcprov-jdk15on-1.60.jar:lib/com.squareup.okio.okio-1.15.0.jar:lib/io.fabric8.kubernetes-model-storageclass-4.12.0.jar:lib/org.keycloak.keycloak-common-11.0.2.jar:lib/io.netty.netty-codec-http2-4.1.50.Final.jar:lib/io.fabric8.kubernetes-model-coordination-4.12.0.jar:lib/io.fabric8.kubernetes-model-settings-4.12.0.jar:lib/io.strimzi.kafka-oauth-client-0.6.1.jar:lib/org.xerial.snappy.snappy-java-1.1.7.3.jar:lib/org.hdrhistogram.HdrHistogram-2.1.11.jar:lib/io.strimzi.operator-common-0.20.1.jar:lib/io.prometheus.simpleclient-0.7.0.jar:lib/com.fasterxml.jackson.module.jackson-module-jaxb-annotations-2.10.4.jar:lib/com.sun.activation.jakarta.activation-1.2.1.jar:lib/io.fabric8.kubernetes-model-events-4.12.0.jar:lib/com.squareup.okhttp3.logging-interceptor-3.12.12.jar:lib/javax.xml.bind.jaxb-api-2.3.0.jar:lib/org.apache.yetus.audience-annotations-0.5.0.jar:lib/io.fabric8.kubernetes-model-discovery-4.12.0.jar:lib/javax.annotation.javax.annotation-api-1.3.2.jar:lib/io.fabric8.kubernetes-model-apiextensions-4.12.0.jar:lib/io.fabric8.kubernetes-client-4.12.0.jar:lib/io.netty.netty-handler-proxy-4.1.50.Final.jar:lib/io.netty.netty-codec-dns-4.1.50.Final.jar:lib/io.micrometer.micrometer-core-1.3.1.jar:lib/io.fabric8.kubernetes-model-certificates-4.12.0.jar:lib/io.fabric8.openshift-model-monitoring-4.12.0.jar:lib/io.strimzi.kafka-oauth-common-0.6.1.jar:lib/jakarta.activation.jakarta.activation-api-1.2.1.jar:lib/io.netty.netty-transport-native-epoll-4.1.50.Final.jar:lib/io.netty.netty-transport-4.1.50.Final.jar:lib/io.strimzi.config-model-0.20.1.jar:lib/io.fabric8.kubernetes-model-metrics-4.12.0.jar:lib/com.github.mifmif.generex-1.0.2.jar:lib/io.fabric8.kubernetes-model-common-4.12.0.jar:lib/io.netty.netty-common-4.1.50.Final.jar:lib/io.strimzi.kafka-oauth-server-0.6.1.jar:lib/io.netty.netty-transport-native-unix-common-4.1.50.Final.jar:lib/io.vertx.vertx-core-3.9.1.jar:lib/com.github.luben.zstd-jni-1.4.4-7.jar:lib/io.netty.netty-resolver-dns-4.1.50.Final.jar:lib/org.yaml.snakeyaml-1.26.jar:lib/io.fabric8.kubernetes-model-core-4.12.0.jar:lib/io.strimzi.certificate-manager-0.20.1.jar:lib/io.fabric8.kubernetes-model-networking-4.12.0.jar:lib/io.fabric8.openshift-model-4.12.0.jar:lib/io.netty.netty-resolver-4.1.50.Final.jar:lib/org.lz4.lz4-java-1.7.1.jar:lib/org.apache.kafka.kafka-clients-2.6.0.jar:lib/io.fabric8.openshift-model-operatorhub-4.12.0.jar:lib/org.apache.logging.log4j.log4j-slf4j-impl-2.13.3.jar:lib/dk.brics.automaton.automaton-1.11-8.jar:lib/io.netty.netty-codec-4.1.50.Final.jar:lib/io.netty.netty-codec-socks-4.1.50.Final.jar:lib/io.strimzi.api-0.20.1.jar:lib/io.fabric8.kubernetes-model-batch-4.12.0.jar:lib/io.fabric8.kubernetes-model-extensions-4.12.0.jar:lib/com.mchange.c3p0-0.9.5.4.jar:lib/com.fasterxml.jackson.core.jackson-databind-2.10.4.jar:lib/jakarta.xml.bind.jakarta.xml.bind-api-2.3.2.jar:lib/io.fabric8.kubernetes-model-scheduling-4.12.0.jar:lib/io.fabric8.zjsonpatch-0.3.0.jar:lib/com.fasterxml.jackson.core.jackson-annotations-2.10.4.jar:lib/com.mchange.mchange-commons-java-0.2.15.jar:lib/io.fabric8.openshift-model-console-4.12.0.jar:lib/com.squareup.okhttp3.okhttp-3.12.6.jar:lib/io.netty.netty-transport-native-epoll-4.1.50.Final-linux-x86_64.jar:lib/com.fasterxml.jackson.dataformat.jackson-dataformat-yaml-2.10.4.jar:lib/io.netty.netty-handler-4.1.50.Final.jar:lib/io.fabric8.kubernetes-model-autoscaling-4.12.0.jar:lib/org.slf4j.slf4j-api-1.7.25.jar:lib/org.latencyutils.LatencyUtils-2.0.3.jar:lib/io.fabric8.kubernetes-model-admissionregistration-4.12.0.jar:lib/io.fabric8.kubernetes-model-policy-4.12.0.jar:lib/io.fabric8.kubernetes-model-rbac-4.12.0.jar:lib/org.apache.zookeeper.zookeeper-jute-3.5.8.jar:lib/io.fabric8.openshift-client-4.12.0.jar:lib/org.bouncycastle.bcpkix-jdk15on-1.65.jar:lib/com.fasterxml.jackson.datatype.jackson-datatype-jsr310-2.11.2.jar:lib/io.netty.netty-buffer-4.1.50.Final.jar:lib/org.apache.logging.log4j.log4j-core-2.13.3.jar:lib/io.netty.netty-codec-http-4.1.50.Final.jar:lib/io.micrometer.micrometer-registry-prometheus-1.3.1.jar:lib/org.quartz-scheduler.quartz-2.3.2.jar io.strimzi.operator.cluster.Main
2021-07-01T16:32:03.5056476Z 2021-07-01 16:29:22 INFO Main:60 - ClusterOperator 0.20.1 is starting
2021-07-01T16:32:03.5057591Z 2021-07-01 16:29:23 DEBUG InternalLoggerFactory:61 - Using SLF4J as the default logging framework
2021-07-01T16:32:03.5058733Z 2021-07-01 16:29:23 DEBUG Config:515 - Trying to configure client from Kubernetes config...
2021-07-01T16:32:03.5059887Z 2021-07-01 16:29:23 DEBUG Config:521 - Did not find Kubernetes config at: [/home/strimzi/.kube/config]. Ignoring.
2021-07-01T16:32:03.5060968Z 2021-07-01 16:29:23 DEBUG Config:447 - Trying to configure client from service account...
2021-07-01T16:32:03.5061922Z 2021-07-01 16:29:23 DEBUG Config:452 - Found service account host and port:
2021-07-01T16:32:03.5063218Z 2021-07-01 16:29:23 DEBUG Config:458 - Found service account ca cert at: [/var/run/secrets/kubernetes.io/serviceaccount/ca.crt].
2021-07-01T16:32:03.5064595Z 2021-07-01 16:29:23 DEBUG Config:465 - Found service account token at: [/var/run/secrets/kubernetes.io/serviceaccount/token].
2021-07-01T16:32:03.5065917Z 2021-07-01 16:29:23 DEBUG Config:720 - Trying to configure client namespace from Kubernetes service account namespace path...
2021-07-01T16:32:03.5067286Z 2021-07-01 16:29:23 DEBUG Config:725 - Found service account namespace at: [/var/run/secrets/kubernetes.io/serviceaccount/namespace].
2021-07-01T16:32:03.5068989Z 2021-07-01 16:29:24 DEBUG PlatformFeaturesAvailability:172 - https:// /apis/route.openshift.io/v1 returned 404. This API is not supported.
2021-07-01T16:32:03.5070808Z 2021-07-01 16:29:24 DEBUG PlatformFeaturesAvailability:172 - https:// /apis/build.openshift.io/v1 returned 404. This API is not supported.
2021-07-01T16:32:03.5072708Z 2021-07-01 16:29:24 DEBUG PlatformFeaturesAvailability:172 - https:// /apis/apps.openshift.io/v1 returned 404. This API is not supported.
2021-07-01T16:32:03.5074504Z 2021-07-01 16:29:24 DEBUG PlatformFeaturesAvailability:172 - https:// /apis/image.openshift.io/v1 returned 404. This API is not supported.
2021-07-01T16:32:03.5076888Z 2021-07-01 16:29:24 INFO Main:80 - Environment facts gathered: ClusterOperatorConfig(KubernetesVersion=1.19,OpenShiftRoutes=false,OpenShiftBuilds=false,OpenShiftImageStreams=false,OpenShiftDeploymentConfigs=false)
2021-07-01T16:32:03.5078620Z 2021-07-01 16:29:24 INFO Util:272 - Using config:
2021-07-01T16:32:03.5079291Z PATH: /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
2021-07-01T16:32:03.5080287Z STRIMZI_DEFAULT_KAFKA_BRIDGE_IMAGE: :443/dkr-all/dkr-io/strimzi/kafka-bridge:0.19.0
2021-07-01T16:32:03.5081015Z STRIMZI_IMAGE_PULL_SECRETS: regcred
2021-07-01T16:32:03.5081953Z TINI_SHA256: 12d20136605531b09a2c2dac02ccee85e1b874eb322ef6baf7561cd93f93c855
2021-07-01T16:32:03.5083642Z STRIMZI_KAFKA_MIRROR_MAKER_IMAGES: 2.5.0= :443/dkr-all/dkr-io/strimzi/kafka:0.20.1-kafka-2.5.0
2021-07-01T16:32:03.5084611Z 2.5.1= :443/dkr-all/dkr-io/strimzi/kafka:0.20.1-kafka-2.5.1
2021-07-01T16:32:03.5085441Z 2.6.0= :443/dkr-all/dkr-io/strimzi/kafka:0.20.1-kafka-2.6.0
2021-07-01T16:32:03.5086179Z
2021-07-01T16:32:03.5086638Z KUBERNETES_PORT: tcp://
2021-07-01T16:32:03.5087669Z STRIMZI_DEFAULT_TOPIC_OPERATOR_IMAGE: :443/dkr-all/dkr-io/strimzi/operator:0.20.1
2021-07-01T16:32:03.5089405Z JAVA_OPTS: -Dlog4j2.configurationFile=file:/opt/strimzi/custom-config/log4j2.properties -Dvertx.cacheDirBase=/tmp -Djava.security.egd=file:/dev/./urandom --illegal-access=deny
2021-07-01T16:32:03.5090708Z STRIMZI_FULL_RECONCILIATION_INTERVAL_MS: 120000
2021-07-01T16:32:03.5091293Z KUBERNETES_SERVICE_HOST:
2021-07-01T16:32:03.5137661Z JAVA_CLASSPATH: lib/io.strimzi.cluster-operator-0.20.1.jar:lib/com.fasterxml.jackson.core.jackson-core-2.10.4.jar:lib/io.prometheus.simpleclient_common-0.7.0.jar:lib/io.vertx.vertx-micrometer-metrics-3.9.1.jar:lib/io.strimzi.crd-annotations-0.20.1.jar:lib/io.fabric8.openshift-model-operator-4.12.0.jar:lib/org.apache.zookeeper.zookeeper-3.5.8.jar:lib/org.apache.logging.log4j.log4j-api-2.13.3.jar:lib/com.zaxxer.HikariCP-java7-2.4.13.jar:lib/org.keycloak.keycloak-core-11.0.2.jar:lib/io.fabric8.kubernetes-model-apps-4.12.0.jar:lib/org.bouncycastle.bcprov-jdk15on-1.60.jar:lib/com.squareup.okio.okio-1.15.0.jar:lib/io.fabric8.kubernetes-model-storageclass-4.12.0.jar:lib/org.keycloak.keycloak-common-11.0.2.jar:lib/io.netty.netty-codec-http2-4.1.50.Final.jar:lib/io.fabric8.kubernetes-model-coordination-4.12.0.jar:lib/io.fabric8.kubernetes-model-settings-4.12.0.jar:lib/io.strimzi.kafka-oauth-client-0.6.1.jar:lib/org.xerial.snappy.snappy-java-1.1.7.3.jar:lib/org.hdrhistogram.HdrHistogram-2.1.11.jar:lib/io.strimzi.operator-common-0.20.1.jar:lib/io.prometheus.simpleclient-0.7.0.jar:lib/com.fasterxml.jackson.module.jackson-module-jaxb-annotations-2.10.4.jar:lib/com.sun.activation.jakarta.activation-1.2.1.jar:lib/io.fabric8.kubernetes-model-events-4.12.0.jar:lib/com.squareup.okhttp3.logging-interceptor-3.12.12.jar:lib/javax.xml.bind.jaxb-api-2.3.0.jar:lib/org.apache.yetus.audience-annotations-0.5.0.jar:lib/io.fabric8.kubernetes-model-discovery-4.12.0.jar:lib/javax.annotation.javax.annotation-api-1.3.2.jar:lib/io.fabric8.kubernetes-model-apiextensions-4.12.0.jar:lib/io.fabric8.kubernetes-client-4.12.0.jar:lib/io.netty.netty-handler-proxy-4.1.50.Final.jar:lib/io.netty.netty-codec-dns-4.1.50.Final.jar:lib/io.micrometer.micrometer-core-1.3.1.jar:lib/io.fabric8.kubernetes-model-certificates-4.12.0.jar:lib/io.fabric8.openshift-model-monitoring-4.12.0.jar:lib/io.strimzi.kafka-oauth-common-0.6.1.jar:lib/jakarta.activation.jakarta.activation-api-1.2.1.jar:lib/io.netty.netty-transport-native-epoll-4.1.50.Final.jar:lib/io.netty.netty-transport-4.1.50.Final.jar:lib/io.strimzi.config-model-0.20.1.jar:lib/io.fabric8.kubernetes-model-metrics-4.12.0.jar:lib/com.github.mifmif.generex-1.0.2.jar:lib/io.fabric8.kubernetes-model-common-4.12.0.jar:lib/io.netty.netty-common-4.1.50.Final.jar:lib/io.strimzi.kafka-oauth-server-0.6.1.jar:lib/io.netty.netty-transport-native-unix-common-4.1.50.Final.jar:lib/io.vertx.vertx-core-3.9.1.jar:lib/com.github.luben.zstd-jni-1.4.4-7.jar:lib/io.netty.netty-resolver-dns-4.1.50.Final.jar:lib/org.yaml.snakeyaml-1.26.jar:lib/io.fabric8.kubernetes-model-core-4.12.0.jar:lib/io.strimzi.certificate-manager-0.20.1.jar:lib/io.fabric8.kubernetes-model-networking-4.12.0.jar:lib/io.fabric8.openshift-model-4.12.0.jar:lib/io.netty.netty-resolver-4.1.50.Final.jar:lib/org.lz4.lz4-java-1.7.1.jar:lib/org.apache.kafka.kafka-clients-2.6.0.jar:lib/io.fabric8.openshift-model-operatorhub-4.12.0.jar:lib/org.apache.logging.log4j.log4j-slf4j-impl-2.13.3.jar:lib/dk.brics.automaton.automaton-1.11-8.jar:lib/io.netty.netty-codec-4.1.50.Final.jar:lib/io.netty.netty-codec-socks-4.1.50.Final.jar:lib/io.strimzi.api-0.20.1.jar:lib/io.fabric8.kubernetes-model-batch-4.12.0.jar:lib/io.fabric8.kubernetes-model-extensions-4.12.0.jar:lib/com.mchange.c3p0-0.9.5.4.jar:lib/com.fasterxml.jackson.core.jackson-databind-2.10.4.jar:lib/jakarta.xml.bind.jakarta.xml.bind-api-2.3.2.jar:lib/io.fabric8.kubernetes-model-scheduling-4.12.0.jar:lib/io.fabric8.zjsonpatch-0.3.0.jar:lib/com.fasterxml.jackson.core.jackson-annotations-2.10.4.jar:lib/com.mchange.mchange-commons-java-0.2.15.jar:lib/io.fabric8.openshift-model-console-4.12.0.jar:lib/com.squareup.okhttp3.okhttp-3.12.6.jar:lib/io.netty.netty-transport-native-epoll-4.1.50.Final-linux-x86_64.jar:lib/com.fasterxml.jackson.dataformat.jackson-dataformat-yaml-2.10.4.jar:lib/io.netty.netty-handler-4.1.50.Final.jar:lib/io.fabric8.kubernetes-model-autoscaling-4.12.0.jar:lib/org.slf4j.slf4j-api-1.7.25.jar:lib/org.latencyutils.LatencyUtils-2.0.3.jar:lib/io.fabric8.kubernetes-model-admissionregistration-4.12.0.jar:lib/io.fabric8.kubernetes-model-policy-4.12.0.jar:lib/io.fabric8.kubernetes-model-rbac-4.12.0.jar:lib/org.apache.zookeeper.zookeeper-jute-3.5.8.jar:lib/io.fabric8.openshift-client-4.12.0.jar:lib/org.bouncycastle.bcpkix-jdk15on-1.65.jar:lib/com.fasterxml.jackson.datatype.jackson-datatype-jsr310-2.11.2.jar:lib/io.netty.netty-buffer-4.1.50.Final.jar:lib/org.apache.logging.log4j.log4j-core-2.13.3.jar:lib/io.netty.netty-codec-http-4.1.50.Final.jar:lib/io.micrometer.micrometer-registry-prometheus-1.3.1.jar:lib/org.quartz-scheduler.quartz-2.3.2.jar
2021-07-01T16:32:03.5180605Z STRIMZI_KAFKA_CONNECT_S2I_IMAGES: 2.5.0= :443/dkr-all/dkr-io/strimzi/kafka:0.20.1-kafka-2.5.0
2021-07-01T16:32:03.5181570Z 2.5.1= :443/dkr-all/dkr-io/strimzi/kafka:0.20.1-kafka-2.5.1
2021-07-01T16:32:03.5182406Z 2.6.0= :443/dkr-all/dkr-io/strimzi/kafka:0.20.1-kafka-2.6.0
2021-07-01T16:32:03.5182923Z
2021-07-01T16:32:03.5183304Z TINI_VERSION: v0.18.0
2021-07-01T16:32:03.5183771Z STRIMZI_HOME: /opt/strimzi
2021-07-01T16:32:03.5184734Z STRIMZI_DEFAULT_TLS_SIDECAR_CRUISE_CONTROL_IMAGE: :443/dkr-all/dkr-io/strimzi/kafka:0.20.1-kafka-2.6.0
2021-07-01T16:32:03.5185400Z PWD: /opt/strimzi
2021-07-01T16:32:03.5186263Z STRIMZI_DEFAULT_CRUISE_CONTROL_IMAGE: :443/dkr-all/dkr-io/strimzi/kafka:0.20.1-kafka-2.6.0
2021-07-01T16:32:03.5187034Z KUBERNETES_PORT_443_TCP: tcp://
2021-07-01T16:32:03.5187616Z STRIMZI_OPERATION_TIMEOUT_MS: 300000
2021-07-01T16:32:03.5188331Z JAVA_MAIN: io.strimzi.operator.cluster.Main
2021-07-01T16:32:03.5189099Z KUBERNETES_PORT_443_TCP_ADDR:
2021-07-01T16:32:03.5189562Z STRIMZI_VERSION: 0.20.1
2021-07-01T16:32:03.5190384Z STRIMZI_DEFAULT_JMXTRANS_IMAGE: :443/dkr-all/dkr-io/strimzi/jmxtrans:0.20.1
2021-07-01T16:32:03.5191175Z STRIMZI_NAMESPACE: gpm-kafka
2021-07-01T16:32:03.5192080Z STRIMZI_KAFKA_MIRROR_MAKER_2_IMAGES: 2.5.0= :443/dkr-all/dkr-io/strimzi/kafka:0.20.1-kafka-2.5.0
2021-07-01T16:32:03.5193135Z 2.5.1= :443/dkr-all/dkr-io/strimzi/kafka:0.20.1-kafka-2.5.1
2021-07-01T16:32:03.5193995Z 2.6.0= :443/dkr-all/dkr-io/strimzi/kafka:0.20.1-kafka-2.6.0
2021-07-01T16:32:03.5194384Z
2021-07-01T16:32:03.5194794Z KUBERNETES_PORT_443_TCP_PROTO: tcp
2021-07-01T16:32:03.5195673Z STRIMZI_DEFAULT_USER_OPERATOR_IMAGE: :443/dkr-all/dkr-io/strimzi/operator:0.20.1
2021-07-01T16:32:03.5196323Z STRIMZI_FEATURE_GATES:
2021-07-01T16:32:03.5196861Z KUBERNETES_SERVICE_PORT: 443
2021-07-01T16:32:03.5197773Z STRIMZI_DEFAULT_KAFKA_EXPORTER_IMAGE: :443/dkr-all/dkr-io/strimzi/kafka:0.20.1-kafka-2.6.0
2021-07-01T16:32:03.5198838Z STRIMZI_DEFAULT_KAFKA_INIT_IMAGE: :443/dkr-all/dkr-io/strimzi/operator:0.20.1
2021-07-01T16:32:03.5199970Z HOSTNAME: strimzi-cluster-operator-644c985f48-ctbrm
2021-07-01T16:32:03.5201386Z STRIMZI_KAFKA_CONNECT_IMAGES: 2.5.0= :443/dkr-all/dkr-io/strimzi/kafka:0.20.1-kafka-2.5.0
2021-07-01T16:32:03.5202338Z 2.5.1= :443/dkr-all/dkr-io/strimzi/kafka:0.20.1-kafka-2.5.1
2021-07-01T16:32:03.5203305Z 2.6.0= :443/dkr-all/dkr-io/strimzi/kafka:0.20.1-kafka-2.6.0
2021-07-01T16:32:03.5203695Z
2021-07-01T16:32:03.5204466Z STRIMZI_KAFKA_IMAGES: 2.5.0= :443/dkr-all/dkr-io/strimzi/kafka:0.20.1-kafka-2.5.0
2021-07-01T16:32:03.5205331Z 2.5.1= :443/dkr-all/dkr-io/strimzi/kafka:0.20.1-kafka-2.5.1
2021-07-01T16:32:03.5206187Z 2.6.0= :443/dkr-all/dkr-io/strimzi/kafka:0.20.1-kafka-2.6.0
2021-07-01T16:32:03.5206861Z
2021-07-01T16:32:03.5207375Z KUBERNETES_PORT_443_TCP_PORT: 443
2021-07-01T16:32:03.5208356Z STRIMZI_DEFAULT_TLS_SIDECAR_ENTITY_OPERATOR_IMAGE: :443/dkr-all/dkr-io/strimzi/kafka:0.20.1-kafka-2.6.0
2021-07-01T16:32:03.5209164Z KUBERNETES_SERVICE_PORT_HTTPS: 443
2021-07-01T16:32:03.5209600Z SHLVL: 0
2021-07-01T16:32:03.5209995Z HOME: /home/strimzi
2021-07-01T16:32:03.5210749Z STRIMZI_OPERATOR_NAMESPACE: gpm-kafka
2021-07-01T16:32:03.5211264Z MALLOC_ARENA_MAX: 2
2021-07-01T16:32:03.5211538Z
2021-07-01T16:32:03.5213018Z 2021-07-01 16:29:25 INFO Main:121 - The KafkaConnectS2I custom resource definition can only be used in environment which supports OpenShift build, image and apps APIs. These APIs do not seem to be supported in this environment.
2021-07-01T16:32:03.5214639Z 2021-07-01 16:29:25 INFO ClusterOperator:81 - Creating ClusterOperator for namespace gpm-kafka
2021-07-01T16:32:03.5215835Z 2021-07-01 16:29:25 INFO ClusterOperator:98 - Starting ClusterOperator for namespace gpm-kafka
2021-07-01T16:32:03.5217885Z 2021-07-01 16:29:25 DEBUG WatchConnectionManager:111 - Connecting websocket ... io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager@42759ba
2021-07-01T16:32:03.5220001Z 2021-07-01 16:29:25 WARN WatchConnectionManager:198 - Exec Failure: HTTP 404, Status: 404 - 404 page not found
2021-07-01T16:32:03.5220577Z
2021-07-01T16:32:03.5221431Z java.net.ProtocolException: Expected HTTP 101 response but was '404 Not Found'
2021-07-01T16:32:03.5223413Z at okhttp3.internal.ws.RealWebSocket.checkResponse(RealWebSocket.java:229) [com.squareup.okhttp3.okhttp-3.12.6.jar:?]
2021-07-01T16:32:03.5225808Z at okhttp3.internal.ws.RealWebSocket$2.onResponse(RealWebSocket.java:196) [com.squareup.okhttp3.okhttp-3.12.6.jar:?]
2021-07-01T16:32:03.5227762Z at okhttp3.RealCall$AsyncCall.execute(RealCall.java:203) [com.squareup.okhttp3.okhttp-3.12.6.jar:?]
2021-07-01T16:32:03.5229723Z at okhttp3.internal.NamedRunnable.run(NamedRunnable.java:32) [com.squareup.okhttp3.okhttp-3.12.6.jar:?]
2021-07-01T16:32:03.5231592Z at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
2021-07-01T16:32:03.5233431Z at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
2021-07-01T16:32:03.5234614Z at java.lang.Thread.run(Thread.java:834) [?:?]
2021-07-01T16:32:03.5235719Z 2021-07-01 16:29:25 DEBUG WatchConnectionManager:300 - Submitting reconnect task to the executor
2021-07-01T16:32:03.5237878Z 2021-07-01 16:29:25 DEBUG WatchConnectionManager:346 - Force closing the watch io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager@42759ba
2021-07-01T16:32:03.5239882Z 2021-07-01 16:29:25 INFO AbstractOperator:442 - Watcher closed in namespace gpm-kafka
2021-07-01T16:32:03.5240948Z 2021-07-01 16:29:25 DEBUG WatchConnectionManager:316 - Scheduling reconnect task
2021-07-01T16:32:03.5242160Z 2021-07-01 16:29:25 DEBUG WatchConnectionManager:388 - Current reconnect backoff is 1000 milliseconds (T0)
2021-07-01T16:32:03.5244545Z 2021-07-01 16:29:25 DEBUG WatchConnectionManager:111 - Connecting websocket ... io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager@267dce97
2021-07-01T16:32:03.5246688Z 2021-07-01 16:29:25 WARN WatchConnectionManager:198 - Exec Failure: HTTP 404, Status: 404 - 404 page not found
2021-07-01T16:32:03.5247261Z
2021-07-01T16:32:03.5248114Z java.net.ProtocolException: Expected HTTP 101 response but was '404 Not Found'
2021-07-01T16:32:03.5250040Z at okhttp3.internal.ws.RealWebSocket.checkResponse(RealWebSocket.java:229) [com.squareup.okhttp3.okhttp-3.12.6.jar:?]
2021-07-01T16:32:03.5252944Z at okhttp3.internal.ws.RealWebSocket$2.onResponse(RealWebSocket.java:196) [com.squareup.okhttp3.okhttp-3.12.6.jar:?]
2021-07-01T16:32:03.5255164Z at okhttp3.RealCall$AsyncCall.execute(RealCall.java:203) [com.squareup.okhttp3.okhttp-3.12.6.jar:?]
2021-07-01T16:32:03.5257259Z at okhttp3.internal.NamedRunnable.run(NamedRunnable.java:32) [com.squareup.okhttp3.okhttp-3.12.6.jar:?]
2021-07-01T16:32:03.5259567Z at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
2021-07-01T16:32:03.5261501Z at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
2021-07-01T16:32:03.5262802Z at java.lang.Thread.run(Thread.java:834) [?:?]
2021-07-01T16:32:03.5264029Z 2021-07-01 16:29:25 DEBUG WatchConnectionManager:300 - Submitting reconnect task to the executor
2021-07-01T16:32:03.5266225Z 2021-07-01 16:29:25 DEBUG WatchConnectionManager:346 - Force closing the watch io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager@267dce97
2021-07-01T16:32:03.5268241Z 2021-07-01 16:29:25 INFO AbstractOperator:442 - Watcher closed in namespace gpm-kafka
2021-07-01T16:32:03.5269372Z 2021-07-01 16:29:25 DEBUG WatchConnectionManager:316 - Scheduling reconnect task
2021-07-01T16:32:03.5270583Z 2021-07-01 16:29:25 DEBUG WatchConnectionManager:388 - Current reconnect backoff is 1000 milliseconds (T0)
2021-07-01T16:32:03.5272835Z 2021-07-01 16:29:25 DEBUG WatchConnectionManager:111 - Connecting websocket ... io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager@2f5a06b7
2021-07-01T16:32:03.5274936Z 2021-07-01 16:29:25 WARN WatchConnectionManager:198 - Exec Failure: HTTP 404, Status: 404 - 404 page not found
2021-07-01T16:32:03.5275502Z
2021-07-01T16:32:03.5276343Z java.net.ProtocolException: Expected HTTP 101 response but was '404 Not Found'
2021-07-01T16:32:03.5278363Z at okhttp3.internal.ws.RealWebSocket.checkResponse(RealWebSocket.java:229) [com.squareup.okhttp3.okhttp-3.12.6.jar:?]
2021-07-01T16:32:03.5280813Z at okhttp3.internal.ws.RealWebSocket$2.onResponse(RealWebSocket.java:196) [com.squareup.okhttp3.okhttp-3.12.6.jar:?]
2021-07-01T16:32:03.5282868Z at okhttp3.RealCall$AsyncCall.execute(RealCall.java:203) [com.squareup.okhttp3.okhttp-3.12.6.jar:?]
2021-07-01T16:32:03.5284809Z at okhttp3.internal.NamedRunnable.run(NamedRunnable.java:32) [com.squareup.okhttp3.okhttp-3.12.6.jar:?]
2021-07-01T16:32:03.5286690Z at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
2021-07-01T16:32:03.5288392Z at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
2021-07-01T16:32:03.5289553Z at java.lang.Thread.run(Thread.java:834) [?:?]
2021-07-01T16:32:03.5290666Z 2021-07-01 16:29:25 DEBUG WatchConnectionManager:300 - Submitting reconnect task to the executor
2021-07-01T16:32:03.5292848Z 2021-07-01 16:29:25 DEBUG WatchConnectionManager:346 - Force closing the watch io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager@2f5a06b7
2021-07-01T16:32:03.5294871Z 2021-07-01 16:29:25 INFO AbstractOperator:442 - Watcher closed in namespace gpm-kafka
2021-07-01T16:32:03.5295999Z 2021-07-01 16:29:25 DEBUG WatchConnectionManager:316 - Scheduling reconnect task
2021-07-01T16:32:03.5297198Z 2021-07-01 16:29:25 DEBUG WatchConnectionManager:388 - Current reconnect backoff is 1000 milliseconds (T0)
2021-07-01T16:32:03.5299365Z 2021-07-01 16:29:25 DEBUG WatchConnectionManager:111 - Connecting websocket ... io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager@348bf832
2021-07-01T16:32:03.5301445Z 2021-07-01 16:29:25 WARN WatchConnectionManager:198 - Exec Failure: HTTP 404, Status: 404 - 404 page not found
2021-07-01T16:32:03.5302012Z
2021-07-01T16:32:03.5302943Z java.net.ProtocolException: Expected HTTP 101 response but was '404 Not Found'
2021-07-01T16:32:03.5304878Z at okhttp3.internal.ws.RealWebSocket.checkResponse(RealWebSocket.java:229) [com.squareup.okhttp3.okhttp-3.12.6.jar:?]
2021-07-01T16:32:03.5307241Z at okhttp3.internal.ws.RealWebSocket$2.onResponse(RealWebSocket.java:196) [com.squareup.okhttp3.okhttp-3.12.6.jar:?]
2021-07-01T16:32:03.5309243Z at okhttp3.RealCall$AsyncCall.execute(RealCall.java:203) [com.squareup.okhttp3.okhttp-3.12.6.jar:?]
2021-07-01T16:32:03.5311148Z at okhttp3.internal.NamedRunnable.run(NamedRunnable.java:32) [com.squareup.okhttp3.okhttp-3.12.6.jar:?]
2021-07-01T16:32:03.5313085Z at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
2021-07-01T16:32:03.5315203Z at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
2021-07-01T16:32:03.5316325Z at java.lang.Thread.run(Thread.java:834) [?:?]
2021-07-01T16:32:03.5317435Z 2021-07-01 16:29:25 DEBUG WatchConnectionManager:300 - Submitting reconnect task to the executor
2021-07-01T16:32:03.5319551Z 2021-07-01 16:29:25 DEBUG WatchConnectionManager:346 - Force closing the watch io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager@348bf832
2021-07-01T16:32:03.5321487Z 2021-07-01 16:29:25 INFO AbstractOperator:442 - Watcher closed in namespace gpm-kafka
2021-07-01T16:32:03.5322785Z 2021-07-01 16:29:25 DEBUG WatchConnectionManager:316 - Scheduling reconnect task
2021-07-01T16:32:03.5324065Z 2021-07-01 16:29:25 DEBUG WatchConnectionManager:388 - Current reconnect backoff is 1000 milliseconds (T0)
2021-07-01T16:32:03.5326159Z 2021-07-01 16:29:25 DEBUG WatchConnectionManager:111 - Connecting websocket ... io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager@28fcd85a
2021-07-01T16:32:03.5328277Z 2021-07-01 16:29:25 WARN WatchConnectionManager:198 - Exec Failure: HTTP 404, Status: 404 - 404 page not found
2021-07-01T16:32:03.5328901Z
2021-07-01T16:32:03.5329752Z java.net.ProtocolException: Expected HTTP 101 response but was '404 Not Found'
2021-07-01T16:32:03.5331607Z at okhttp3.internal.ws.RealWebSocket.checkResponse(RealWebSocket.java:229) [com.squareup.okhttp3.okhttp-3.12.6.jar:?]
2021-07-01T16:32:03.5334093Z at okhttp3.internal.ws.RealWebSocket$2.onResponse(RealWebSocket.java:196) [com.squareup.okhttp3.okhttp-3.12.6.jar:?]
2021-07-01T16:32:03.5336005Z at okhttp3.RealCall$AsyncCall.execute(RealCall.java:203) [com.squareup.okhttp3.okhttp-3.12.6.jar:?]
2021-07-01T16:32:03.5337876Z at okhttp3.internal.NamedRunnable.run(NamedRunnable.java:32) [com.squareup.okhttp3.okhttp-3.12.6.jar:?]
2021-07-01T16:32:03.5339778Z at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
2021-07-01T16:32:03.5341462Z at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
2021-07-01T16:32:03.5342717Z at java.lang.Thread.run(Thread.java:834) [?:?]
2021-07-01T16:32:03.5343822Z 2021-07-01 16:29:25 DEBUG WatchConnectionManager:300 - Submitting reconnect task to the executor
2021-07-01T16:32:03.5345947Z 2021-07-01 16:29:25 DEBUG WatchConnectionManager:346 - Force closing the watch io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager@28fcd85a
2021-07-01T16:32:03.5347902Z 2021-07-01 16:29:25 INFO AbstractOperator:442 - Watcher closed in namespace gpm-kafka
2021-07-01T16:32:03.5348998Z 2021-07-01 16:29:25 DEBUG WatchConnectionManager:316 - Scheduling reconnect task
2021-07-01T16:32:03.5350204Z 2021-07-01 16:29:25 DEBUG WatchConnectionManager:388 - Current reconnect backoff is 1000 milliseconds (T0)
2021-07-01T16:32:03.5352343Z 2021-07-01 16:29:25 DEBUG WatchConnectionManager:111 - Connecting websocket ... io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager@3afc76c6
2021-07-01T16:32:03.5354565Z 2021-07-01 16:29:25 WARN WatchConnectionManager:198 - Exec Failure: HTTP 404, Status: 404 - 404 page not found
2021-07-01T16:32:03.5355140Z
2021-07-01T16:32:03.5355982Z java.net.ProtocolException: Expected HTTP 101 response but was '404 Not Found'
2021-07-01T16:32:03.5357847Z at okhttp3.internal.ws.RealWebSocket.checkResponse(RealWebSocket.java:229) [com.squareup.okhttp3.okhttp-3.12.6.jar:?]
2021-07-01T16:32:03.5360197Z at okhttp3.internal.ws.RealWebSocket$2.onResponse(RealWebSocket.java:196) [com.squareup.okhttp3.okhttp-3.12.6.jar:?]
2021-07-01T16:32:03.5362112Z at okhttp3.RealCall$AsyncCall.execute(RealCall.java:203) [com.squareup.okhttp3.okhttp-3.12.6.jar:?]
2021-07-01T16:32:03.5364130Z at okhttp3.internal.NamedRunnable.run(NamedRunnable.java:32) [com.squareup.okhttp3.okhttp-3.12.6.jar:?]
2021-07-01T16:32:03.5365950Z at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
2021-07-01T16:32:03.5367939Z at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
2021-07-01T16:32:03.5369246Z at java.lang.Thread.run(Thread.java:834) [?:?]
2021-07-01T16:32:03.5370359Z 2021-07-01 16:29:25 DEBUG WatchConnectionManager:300 - Submitting reconnect task to the executor
2021-07-01T16:32:03.5372544Z 2021-07-01 16:29:25 DEBUG WatchConnectionManager:346 - Force closing the watch io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager@3afc76c6
2021-07-01T16:32:03.5374558Z 2021-07-01 16:29:25 DEBUG WatchConnectionManager:316 - Scheduling reconnect task
2021-07-01T16:32:03.5375800Z 2021-07-01 16:29:25 DEBUG WatchConnectionManager:388 - Current reconnect backoff is 1000 milliseconds (T0)
2021-07-01T16:32:03.5377909Z 2021-07-01 16:29:25 DEBUG WatchConnectionManager:111 - Connecting websocket ... io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager@c84a66f
2021-07-01T16:32:03.5380012Z 2021-07-01 16:29:25 WARN WatchConnectionManager:198 - Exec Failure: HTTP 404, Status: 404 - 404 page not found
2021-07-01T16:32:03.5380581Z
2021-07-01T16:32:03.5381477Z java.net.ProtocolException: Expected HTTP 101 response but was '404 Not Found'
2021-07-01T16:32:03.5383458Z at okhttp3.internal.ws.RealWebSocket.checkResponse(RealWebSocket.java:229) [com.squareup.okhttp3.okhttp-3.12.6.jar:?]
2021-07-01T16:32:03.5385772Z at okhttp3.internal.ws.RealWebSocket$2.onResponse(RealWebSocket.java:196) [com.squareup.okhttp3.okhttp-3.12.6.jar:?]
2021-07-01T16:32:03.5387687Z at okhttp3.RealCall$AsyncCall.execute(RealCall.java:203) [com.squareup.okhttp3.okhttp-3.12.6.jar:?]
2021-07-01T16:32:03.5389741Z at okhttp3.internal.NamedRunnable.run(NamedRunnable.java:32) [com.squareup.okhttp3.okhttp-3.12.6.jar:?]
2021-07-01T16:32:03.5391607Z at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
2021-07-01T16:32:03.5393417Z at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
2021-07-01T16:32:03.5394561Z at java.lang.Thread.run(Thread.java:834) [?:?]
2021-07-01T16:32:03.5395684Z 2021-07-01 16:29:25 DEBUG WatchConnectionManager:300 - Submitting reconnect task to the executor
2021-07-01T16:32:03.5397731Z 2021-07-01 16:29:25 DEBUG WatchConnectionManager:346 - Force closing the watch io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager@c84a66f
2021-07-01T16:32:03.5399762Z 2021-07-01 16:29:25 DEBUG WatchConnectionManager:316 - Scheduling reconnect task
2021-07-01T16:32:03.5401172Z 2021-07-01 16:29:25 DEBUG WatchConnectionManager:388 - Current reconnect backoff is 1000 milliseconds (T0)
2021-07-01T16:32:03.5402372Z 2021-07-01 16:29:25 ERROR Main:156 - Cluster Operator verticle in namespace gpm-kafka failed to start
2021-07-01T16:32:03.5403858Z io.fabric8.kubernetes.client.KubernetesClientException: 404 page not found
2021-07-01T16:32:03.5404785Z
2021-07-01T16:32:03.5406884Z at io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager$1.onFailure(WatchConnectionManager.java:202) ~[io.fabric8.kubernetes-client-4.12.0.jar:?]
2021-07-01T16:32:03.5409931Z at okhttp3.internal.ws.RealWebSocket.failWebSocket(RealWebSocket.java:571) ~[com.squareup.okhttp3.okhttp-3.12.6.jar:?]
2021-07-01T16:32:03.5412262Z at okhttp3.internal.ws.RealWebSocket$2.onResponse(RealWebSocket.java:198) ~[com.squareup.okhttp3.okhttp-3.12.6.jar:?]
2021-07-01T16:32:03.5414299Z at okhttp3.RealCall$AsyncCall.execute(RealCall.java:203) ~[com.squareup.okhttp3.okhttp-3.12.6.jar:?]
2021-07-01T16:32:03.5416176Z at okhttp3.internal.NamedRunnable.run(NamedRunnable.java:32) ~[com.squareup.okhttp3.okhttp-3.12.6.jar:?]
2021-07-01T16:32:03.5418006Z at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
2021-07-01T16:32:03.5420173Z at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
2021-07-01T16:32:03.5421287Z at java.lang.Thread.run(Thread.java:834) [?:?]
2021-07-01T16:32:03.5421983Z Suppressed: java.lang.Throwable: waiting here
2021-07-01T16:32:03.5424110Z at io.fabric8.kubernetes.client.utils.Utils.waitUntilReady(Utils.java:154) ~[io.fabric8.kubernetes-client-4.12.0.jar:?]
2021-07-01T16:32:03.5428079Z at io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager.waitUntilReady(WatchConnectionManager.java:341) ~[io.fabric8.kubernetes-client-4.12.0.jar:?]
2021-07-01T16:32:03.5431846Z at io.fabric8.kubernetes.client.dsl.base.BaseOperation.watch(BaseOperation.java:818) ~[io.fabric8.kubernetes-client-4.12.0.jar:?]
2021-07-01T16:32:03.5434894Z at io.fabric8.kubernetes.client.dsl.base.BaseOperation.watch(BaseOperation.java:791) ~[io.fabric8.kubernetes-client-4.12.0.jar:?]
2021-07-01T16:32:03.5437817Z at io.fabric8.kubernetes.client.dsl.base.BaseOperation.watch(BaseOperation.java:84) ~[io.fabric8.kubernetes-client-4.12.0.jar:?]
2021-07-01T16:32:03.5441842Z at io.strimzi.operator.common.operator.resource.AbstractWatchableResourceOperator.watch(AbstractWatchableResourceOperator.java:63) ~[io.strimzi.operator-common-0.20.1.jar:0.20.1]
2021-07-01T16:32:03.5445838Z at io.strimzi.operator.common.AbstractOperator.lambda$createWatch$16(AbstractOperator.java:431) ~[io.strimzi.operator-common-0.20.1.jar:0.20.1]
2021-07-01T16:32:03.5448485Z at io.strimzi.operator.common.Util.lambda$async$0(Util.java:53) ~[io.strimzi.operator-common-0.20.1.jar:0.20.1]
2021-07-01T16:32:03.5450511Z at io.vertx.core.impl.ContextImpl.lambda$executeBlocking$2(ContextImpl.java:313) ~[io.vertx.vertx-core-3.9.1.jar:3.9.1]
2021-07-01T16:32:03.5452293Z at io.vertx.core.impl.TaskQueue.run(TaskQueue.java:76) ~[io.vertx.vertx-core-3.9.1.jar:3.9.1]
2021-07-01T16:32:03.5454085Z at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
2021-07-01T16:32:03.5455739Z at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
2021-07-01T16:32:03.5458128Z at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [io.netty.netty-common-4.1.50.Final.jar:4.1.50.Final]
2021-07-01T16:32:03.5459893Z at java.lang.Thread.run(Thread.java:834) [?:?]

How to reproduce it (as minimally and precisely as possible):
Anything else we need to know?:
Environment:
cluster_autoscaling = {
"autoscaling_profile": "BALANCED",
"enabled": true,
"max_cpu_cores": 96,
"max_memory_gb": 384,
"min_cpu_cores": 48,
"min_memory_gb": 192
}

Cloud provider or hardware configuration:
GCP - GKE
Hardware disk_type = "pd-standard"
machine_type = "e2-standard-16"
image_type = "COS"
OS (e.g: cat /etc/os-release): Container-Optimized OS with Docker (cos): The cos image uses the Docker container runtime.
Kernel (e.g. uname -a):
Install tools: helm chart from helm repo
Network plugin and version (if this is a network-related bug): We are on VPC subnet inside to organization
Others:
I am a begineer, but is curious to know : Lwe are on GKE, why is the JAVA plugins for OPEN-SHIFT being used in the LOG?
2021-07-01T16:32:03.5068989Z 2021-07-01 16:29:24 DEBUG PlatformFeaturesAvailability:172 - https:// /apis/route.openshift.io/v1 returned 404. This API is not supported.
2021-07-01T16:32:03.5070808Z 2021-07-01 16:29:24 DEBUG PlatformFeaturesAvailability:172 - https:// /apis/build.openshift.io/v1 returned 404. This API is not supported.
2021-07-01T16:32:03.5072708Z 2021-07-01 16:29:24 DEBUG PlatformFeaturesAvailability:172 - https:// /apis/apps.openshift.io/v1 returned 404. This API is not supported.
2021-07-01T16:32:03.5074504Z 2021-07-01 16:29:24 DEBUG PlatformFeaturesAvailability:172 - https:// /apis/image.openshift.io/v1 returned 404. This API is not supported.

is this version do not support GKE?

scholzj · 2021-07-01T18:45:07Z

scholzj
Jul 1, 2021
Maintainer

Formatting the logs as a code would make a bit more readable. But if you read the full log, it says that the OpenShift APIs are not present. It autodetects the platform and decides it is not running on OpenShift - so this is correct if you are on GKE. It does not load any OpenShift plugins.

The 404 error you get (in the actual error which caused the crash loop and not in the OpenShift API detection):

2021-07-01T16:32:03.5217885Z 2021-07-01 16:29:25 DEBUG WatchConnectionManager:111 - Connecting websocket ... io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager@42759ba
2021-07-01T16:32:03.5220001Z 2021-07-01 16:29:25 WARN WatchConnectionManager:198 - Exec Failure: HTTP 404, Status: 404 - 404 page not found
2021-07-01T16:32:03.5220577Z
2021-07-01T16:32:03.5221431Z java.net.ProtocolException: Expected HTTP 101 response but was '404 Not Found'
2021-07-01T16:32:03.5223413Z at okhttp3.internal.ws.RealWebSocket.checkResponse(RealWebSocket.java:229) [com.squareup.okhttp3.okhttp-3.12.6.jar:?]
2021-07-01T16:32:03.5225808Z at okhttp3.internal.ws.RealWebSocket$2.onResponse(RealWebSocket.java:196) [com.squareup.okhttp3.okhttp-3.12.6.jar:?]
2021-07-01T16:32:03.5227762Z at okhttp3.RealCall$AsyncCall.execute(RealCall.java:203) [com.squareup.okhttp3.okhttp-3.12.6.jar:?]
2021-07-01T16:32:03.5229723Z at okhttp3.internal.NamedRunnable.run(NamedRunnable.java:32) [com.squareup.okhttp3.okhttp-3.12.6.jar:?]
2021-07-01T16:32:03.5231592Z at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
2021-07-01T16:32:03.5233431Z at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
2021-07-01T16:32:03.5234614Z at java.lang.Thread.run(Thread.java:834) [?:?]
2021-07-01T16:32:03.5235719Z 2021-07-01 16:29:25 DEBUG WatchConnectionManager:300 - Submitting reconnect task to the executor
2021-07-01T16:32:03.5237878Z 2021-07-01 16:29:25 DEBUG WatchConnectionManager:346 - Force closing the watch io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager@42759ba
2021-07-01T16:32:03.5239882Z 2021-07-01 16:29:25 INFO AbstractOperator:442 - Watcher closed in namespace gpm-kafka
2021-07-01T16:32:03.5240948Z 2021-07-01 16:29:25 DEBUG WatchConnectionManager:316 - Scheduling reconnect task
2021-07-01T16:32:03.5242160Z 2021-07-01 16:29:25 DEBUG WatchConnectionManager:388 - Current reconnect backoff is 1000 milliseconds (T0)

means that you either do not have the CRDs installed (or do not have the right version installed). So you should check the CRDs, make sure all of them are installed and have the right versions (for Strimzi 0.20 that would be a mix of v1alpha1 and v1beta1 versions).

3 replies

rblaine95 Jul 7, 2021

Tossing my hat into the ring

We're currently on Strimzi Kafka Operator 0.19.0, installed with Helm

» helm version
version.BuildInfo{Version:"v3.6.2", GitCommit:"ee407bdf364942bcb8e8c665f82e15aa28009b71", GitTreeState:"clean", GoVersion:"go1.16.5"}

» helm upgrade --install strimzi -n operators strimzi/strimzi-kafka-operator --version=0.19.0

CRDs installed perfectly fine.
But, when we try to upgrade to 0.20.0 or any higher, all the CRDs suddenly get deleted and don't come back

Edit:
Was there a change in the Helm3 chart between 0.19 and 0.20 that moved CRDs out of Helm control?

Edit 2:
This is the 0.20 upgrade command, maybe I'm just being stupid and missing a flag?

» helm upgrade --install strimzi -n operators strimzi/strimzi-kafka-operator --version=0.20.0

Edit 3:
Forgot to add information about what version of Kubernetes

» kubectl version --short
Client Version: v1.21.2
Server Version: v1.16.15-eks-cf75c9

Edit 4: Just spotted #3877

scholzj Jul 7, 2021
Maintainer

I do not remember the versions out of my head. But I guess the 0.19 might have been Helm 2 and the 0.20 Helm 3? They changed where the CRDs should be placed and how should they be installed. But TBH I'm not a Helm user, so I do not know much more.

rblaine95 Jul 8, 2021

I scanned through a couple issues and other discussions here and came across this workaround

Tried it out in our dev environment which doesn't have anything important (because it's dev... why would you have important things in a dev environment?) and successfully upgraded from 0.19 to 0.20 with no problems

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Strimzi

Operator not starting - "Back-off restarting failed container" #5241

Uh oh!

{{title}}

Uh oh!

Replies: 1 comment 3 replies

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{editor}}'s edit

{{editor}}'s edit

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Select a reply

Uh oh!

Strimzi

Operator not starting - "Back-off restarting failed container" #5241

Uh oh!

sreejesh123 Jul 1, 2021

Replies: 1 comment · 3 replies

Uh oh!

scholzj Jul 1, 2021 Maintainer

Uh oh!

Uh oh!

rblaine95 Jul 7, 2021

Uh oh!

scholzj Jul 7, 2021 Maintainer

Uh oh!

rblaine95 Jul 8, 2021

sreejesh123
Jul 1, 2021

Replies: 1 comment 3 replies

scholzj
Jul 1, 2021
Maintainer

scholzj Jul 7, 2021
Maintainer