Error while upgrading Strimzi operator from 0.23.0 to 0.25.0

[KasturiHima]

We are trying to upgrade Strimzi operator from version 0.23.0 to 0.25.0.

When we tried to upgrade from 0.23.0 we got below errors:

kubectl replace -f install/cluster-operator -n databus

serviceaccount/strimzi-cluster-operator replaced

clusterrole.rbac.authorization.k8s.io/strimzi-cluster-operator-namespaced replaced

rolebinding.rbac.authorization.k8s.io/strimzi-cluster-operator replaced

clusterrole.rbac.authorization.k8s.io/strimzi-cluster-operator-global replaced

clusterrolebinding.rbac.authorization.k8s.io/strimzi-cluster-operator replaced

clusterrole.rbac.authorization.k8s.io/strimzi-kafka-broker replaced

clusterrolebinding.rbac.authorization.k8s.io/strimzi-cluster-operator-kafka-broker-delegation replaced

clusterrole.rbac.authorization.k8s.io/strimzi-entity-operator replaced

rolebinding.rbac.authorization.k8s.io/strimzi-cluster-operator-entity-operator-delegation replaced

clusterrole.rbac.authorization.k8s.io/strimzi-kafka-client replaced

clusterrolebinding.rbac.authorization.k8s.io/strimzi-cluster-operator-kafka-client-delegation replaced

customresourcedefinition.apiextensions.k8s.io/kafkatopics.kafka.strimzi.io replaced

customresourcedefinition.apiextensions.k8s.io/kafkausers.kafka.strimzi.io replaced

configmap/strimzi-cluster-operator replaced

Error from server (Invalid): error when replacing "install/cluster-operator/040-Crd-kafka.yaml": CustomResourceDefinition.apiextensions.k8s.io "kafkas.kafka.strimzi.io" is invalid: status.storedVersions[0]: Invalid value: "v1beta1": must appear in spec.versions

Error from server (Invalid): error when replacing "install/cluster-operator/041-Crd-kafkaconnect.yaml": CustomResourceDefinition.apiextensions.k8s.io "kafkaconnects.kafka.strimzi.io" is invalid: status.storedVersions[0]: Invalid value: "v1beta1": must appear in spec.versions

Error from server (Invalid): error when replacing "install/cluster-operator/045-Crd-kafkamirrormaker.yaml": CustomResourceDefinition.apiextensions.k8s.io "kafkamirrormakers.kafka.strimzi.io" is invalid: status.storedVersions[0]: Invalid value: "v1beta1": must appear in spec.versions

Error from server (Invalid): error when replacing "install/cluster-operator/046-Crd-kafkabridge.yaml": CustomResourceDefinition.apiextensions.k8s.io "kafkabridges.kafka.strimzi.io" is invalid: status.storedVersions[0]: Invalid value: "v1alpha1": must appear in spec.versions

Error from server (Invalid): error when replacing "install/cluster-operator/047-Crd-kafkaconnector.yaml": CustomResourceDefinition.apiextensions.k8s.io "kafkaconnectors.kafka.strimzi.io" is invalid: status.storedVersions[0]: Invalid value: "v1alpha1": must appear in spec.versions

Error from server (Invalid): error when replacing "install/cluster-operator/048-Crd-kafkamirrormaker2.yaml": CustomResourceDefinition.apiextensions.k8s.io "kafkamirrormaker2s.kafka.strimzi.io" is invalid: status.storedVersions[0]: Invalid value: "v1alpha1": must appear in spec.versions

Error from server (Invalid): error when replacing "install/cluster-operator/049-Crd-kafkarebalance.yaml": CustomResourceDefinition.apiextensions.k8s.io "kafkarebalances.kafka.strimzi.io" is invalid: status.storedVersions[0]: Invalid value: "v1alpha1": must appear in spec.versions

We modified files with errors and assigned

1. spec.versions = "v1beta1" files files 040-Crd-kafka.yaml, 041-Crd-kafkaconnect.yaml, 045-Crd-kafkamirrormaker.yaml
2. spec.versions = "v1alpha1" in files 046-Crd-kafkabridge.yaml, 047-Crd-kafkaconnector.yaml, 048-Crd-kafkamirrormaker2.yaml, 049-Crd-kafkarebalance.yaml

After making above changes when we try to upgrade the operator the strimzi operator pod goes in "Error" state and is giving below errors in the logs
We tried to Delete the strimzi operator deployment & Delete all strimzi CRD's and created the operator from scratch with "kubectl apply -f install/cluster-operator -n databus"
but still we get the same error message:

2021-08-26 06:28:52 INFO ClusterOperator:94 - Starting ClusterOperator for namespace databus
2021-08-26 06:28:52 WARN WatcherWebSocketListener:73 - Exec Failure: HTTP 404, Status: 404 - 404 page not found

2021-08-26 06:28:52 WARN WatcherWebSocketListener:73 - Exec Failure: HTTP 404, Status: 404 - 404 page not found

2021-08-26 06:28:52 WARN WatcherWebSocketListener:73 - Exec Failure: HTTP 404, Status: 404 - 404 page not found

2021-08-26 06:28:52 WARN WatcherWebSocketListener:73 - Exec Failure: HTTP 404, Status: 404 - 404 page not found

2021-08-26 06:28:52 WARN WatcherWebSocketListener:73 - Exec Failure: HTTP 404, Status: 404 - 404 page not found

2021-08-26 06:28:52 WARN WatcherWebSocketListener:73 - Exec Failure: HTTP 404, Status: 404 - 404 page not found

2021-08-26 06:28:52 WARN WatcherWebSocketListener:73 - Exec Failure: HTTP 404, Status: 404 - 404 page not found

2021-08-26 06:28:52 ERROR Main:153 - Cluster Operator verticle in namespace databus failed to start
io.fabric8.kubernetes.client.KubernetesClientException: 404 page not found

    at io.fabric8.kubernetes.client.dsl.internal.WatcherWebSocketListener.onFailure(WatcherWebSocketListener.java:74) ~[io.fabric8.kubernetes-client-5.6.0.jar:?]
    at okhttp3.internal.ws.RealWebSocket.failWebSocket(RealWebSocket.java:571) ~[com.squareup.okhttp3.okhttp-3.12.6.jar:?]
    at okhttp3.internal.ws.RealWebSocket$2.onResponse(RealWebSocket.java:198) ~[com.squareup.okhttp3.okhttp-3.12.6.jar:?]
    at okhttp3.RealCall$AsyncCall.execute(RealCall.java:203) ~[com.squareup.okhttp3.okhttp-3.12.6.jar:?]
    at okhttp3.internal.NamedRunnable.run(NamedRunnable.java:32) ~[com.squareup.okhttp3.okhttp-3.12.6.jar:?]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
    at java.lang.Thread.run(Thread.java:829) [?:?]
    Suppressed: java.lang.Throwable: waiting here
            at io.fabric8.kubernetes.client.utils.Utils.waitUntilReady(Utils.java:164) ~[io.fabric8.kubernetes-client-5.6.0.jar:?]
            at io.fabric8.kubernetes.client.utils.Utils.waitUntilReadyOrFail(Utils.java:175) ~[io.fabric8.kubernetes-client-5.6.0.jar:?]
            at io.fabric8.kubernetes.client.dsl.internal.WatcherWebSocketListener.waitUntilReady(WatcherWebSocketListener.java:120) ~[io.fabric8.kubernetes-client-5.6.0.jar:?]
            at io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager.waitUntilReady(WatchConnectionManager.java:80) ~[io.fabric8.kubernetes-client-5.6.0.jar:?]
            at io.fabric8.kubernetes.client.dsl.base.BaseOperation.watch(BaseOperation.java:702) ~[io.fabric8.kubernetes-client-5.6.0.jar:?]
            at io.fabric8.kubernetes.client.dsl.base.BaseOperation.watch(BaseOperation.java:675) ~[io.fabric8.kubernetes-client-5.6.0.jar:?]
            at io.fabric8.kubernetes.client.dsl.base.BaseOperation.watch(BaseOperation.java:88) ~[io.fabric8.kubernetes-client-5.6.0.jar:?]
            at io.strimzi.operator.common.operator.resource.AbstractWatchableResourceOperator.watch(AbstractWatchableResourceOperator.java:61) ~[io.strimzi.operator-common-0.25.0.jar:0.25.0]
            at io.strimzi.operator.common.AbstractOperator.lambda$createWatch$18(AbstractOperator.java:440) ~[io.strimzi.operator-common-0.25.0.jar:0.25.0]
            at io.strimzi.operator.common.Util.lambda$async$0(Util.java:65) ~[io.strimzi.operator-common-0.25.0.jar:0.25.0]
            at io.vertx.core.impl.ContextImpl.lambda$null$0(ContextImpl.java:160) ~[io.vertx.vertx-core-4.1.2.jar:4.1.2]
            at io.vertx.core.impl.AbstractContext.dispatch(AbstractContext.java:96) ~[io.vertx.vertx-core-4.1.2.jar:4.1.2]
            at io.vertx.core.impl.ContextImpl.lambda$executeBlocking$1(ContextImpl.java:158) ~[io.vertx.vertx-core-4.1.2.jar:4.1.2]
            at io.vertx.core.impl.TaskQueue.run(TaskQueue.java:76) ~[io.vertx.vertx-core-4.1.2.jar:4.1.2]
            at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
            at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
            at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [io.netty.netty-common-4.1.66.Final.jar:4.1.66.Final]
            at java.lang.Thread.run(Thread.java:829) [?:?]

Expected behavior
Strimzi operator pod should be upgraded from 0.23.0 to 0.25.0 and operator pod should restart without error.
The operator pod is currently going into error state with the above mentioned errors.

Environment (please complete the following information):

• Strimzi version: 0.23.0
• Installation method: https://github.com/strimzi/strimzi-kafka-operator/releases
• Kubernetes cluster: Kubernetes 1.20
• Infrastructure: private cloud[bare metal]

[scholzj]

I'm not sure how you installed 0.23, but you haven't done it properly. And haven't done the CRD upgrade: https://strimzi.io/docs/operators/0.24.0/deploying.html#assembly-upgrade-resources-str ... so you have to do it now.

[misurin]

Hi @scholzj, I'm at 0.22.1 and CRD version is still v1beta1, now I'm looking out t upgrade the CRD to v1beta2 to be able to get the 0.25.0 using this guideline.

According to the guideline, the procedure is:

1. Upgrade all resources.
2. Upgrade the CRDs

However, when I tried to upgrade the resources with the YAML generated by the api-convertion.sh convert-file, K8S won't allow because no match resource version.

Warning: kubectl apply should be used on resource created by either kubectl create --save-config or kubectl apply
configmap/dev-kafka-jmx-exporter-configuration configured
Warning: kubectl apply should be used on resource created by either kubectl create --save-config or kubectl apply
configmap/dev-zookeeper-jmx-exporter-configuration configured
error: unable to recognize "new-dev-kafka.yaml": no matches for kind "Kafka" in version "kafka.strimzi.io/v1beta2"

If I tried to upgrade the CRDs first, I got the following error:

$ bin/api-conversion.sh crd
Changing stored version in all Strimzi CRDs to v1beta2:
Updating KafkaBridge CRD
io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: PATCH at: https://******************.yl4.ap-southeast-1.eks.amazonaws.com/apis/apiextensions.k8s.io/v1beta1/customresourcedefinitions/kafkabridges.kafka.strimzi.io. Message: CustomResourceDefinition.apiextensions.k8s.io "kafkabridges.kafka.strimzi.io" is invalid: spec.versions: Invalid value: []apiextensions.CustomResourceDefinitionVersion{apiextensions.CustomResourceDefinitionVersion{Name:"v1alpha1", Served:true, Storage:false, Schema:(*apiextensions.CustomResourceValidation)(nil), Subresources:(*apiextensions.CustomResourceSubresources)(nil), AdditionalPrinterColumns:[]apiextensions.CustomResourceColumnDefinition(nil)}}: must have exactly one version marked as storage version. Received status: Status(apiVersion=v1, code=422, details=StatusDetails(causes=[StatusCause(field=spec.versions, message=Invalid value: []apiextensions.CustomResourceDefinitionVersion{apiextensions.CustomResourceDefinitionVersion{Name:"v1alpha1", Served:true, Storage:false, Schema:(*apiextensions.CustomResourceValidation)(nil), Subresources:(*apiextensions.CustomResourceSubresources)(nil), AdditionalPrinterColumns:[]apiextensions.CustomResourceColumnDefinition(nil)}}: must have exactly one version marked as storage version, reason=FieldValueInvalid, additionalProperties={})], group=apiextensions.k8s.io, kind=CustomResourceDefinition, name=kafkabridges.kafka.strimzi.io, retryAfterSeconds=null, uid=null, additionalProperties={}), kind=Status, message=CustomResourceDefinition.apiextensions.k8s.io "kafkabridges.kafka.strimzi.io" is invalid: spec.versions: Invalid value: []apiextensions.CustomResourceDefinitionVersion{apiextensions.CustomResourceDefinitionVersion{Name:"v1alpha1", Served:true, Storage:false, Schema:(*apiextensions.CustomResourceValidation)(nil), Subresources:(*apiextensions.CustomResourceSubresources)(nil), AdditionalPrinterColumns:[]apiextensions.CustomResourceColumnDefinition(nil)}}: must have exactly one version marked as storage version, metadata=ListMeta(_continue=null, remainingItemCount=null, resourceVersion=null, selfLink=null, additionalProperties={}), reason=Invalid, status=Failure, additionalProperties={}).
	at io.fabric8.kubernetes.client.dsl.base.OperationSupport.requestFailure(OperationSupport.java:570)
	at io.fabric8.kubernetes.client.dsl.base.OperationSupport.assertResponseCode(OperationSupport.java:509)
	at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:474)
	at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:435)
	at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handlePatch(OperationSupport.java:312)
	at io.fabric8.kubernetes.client.dsl.base.BaseOperation.handlePatch(BaseOperation.java:881)
	at io.fabric8.kubernetes.client.dsl.base.HasMetadataOperation.lambda$patch$1(HasMetadataOperation.java:121)
	at io.fabric8.kubernetes.client.dsl.base.HasMetadataOperation.patch(HasMetadataOperation.java:126)
	at io.fabric8.kubernetes.client.dsl.base.HasMetadataOperation.patch(HasMetadataOperation.java:30)
	at io.strimzi.kafka.api.conversion.cli.CrdUpgradeCommand.changeStoredVersionInSpecForKind(CrdUpgradeCommand.java:100)
	at io.strimzi.kafka.api.conversion.cli.CrdUpgradeCommand.changeStoredVersionInSpec(CrdUpgradeCommand.java:112)
	at io.strimzi.kafka.api.conversion.cli.CrdUpgradeCommand.run(CrdUpgradeCommand.java:162)
	at picocli.CommandLine.executeUserObject(CommandLine.java:1919)
	at picocli.CommandLine.access$1200(CommandLine.java:145)
	at picocli.CommandLine$RunLast.executeUserObjectOfLastSubcommandWithSameParent(CommandLine.java:2332)
	at picocli.CommandLine$RunLast.handle(CommandLine.java:2326)
	at picocli.CommandLine$RunLast.handle(CommandLine.java:2291)
	at picocli.CommandLine$AbstractParseResultHandler.execute(CommandLine.java:2159)
	at picocli.CommandLine.execute(CommandLine.java:2058)
	at io.strimzi.kafka.api.conversion.cli.EntryCommand.main(EntryCommand.java:31)

I'm really not sure if I missed any step?

[misurin]

I have replaced the CRDs from https://github.com/strimzi/strimzi-kafka-operator/tree/0.22.1/helm-charts/helm3/strimzi-kafka-operator/crds:

~/Projects/others/strimzi-kafka-operator/helm-charts/helm3/strimzi-kafka-operator on 05bb5bad8 
$ kubectl replace -f crds
customresourcedefinition.apiextensions.k8s.io/kafkas.kafka.strimzi.io replaced
customresourcedefinition.apiextensions.k8s.io/kafkaconnects.kafka.strimzi.io replaced
customresourcedefinition.apiextensions.k8s.io/kafkaconnects2is.kafka.strimzi.io replaced
customresourcedefinition.apiextensions.k8s.io/kafkatopics.kafka.strimzi.io replaced
customresourcedefinition.apiextensions.k8s.io/kafkausers.kafka.strimzi.io replaced
customresourcedefinition.apiextensions.k8s.io/kafkamirrormakers.kafka.strimzi.io replaced
customresourcedefinition.apiextensions.k8s.io/kafkabridges.kafka.strimzi.io replaced
customresourcedefinition.apiextensions.k8s.io/kafkaconnectors.kafka.strimzi.io replaced
customresourcedefinition.apiextensions.k8s.io/kafkamirrormaker2s.kafka.strimzi.io replaced
customresourcedefinition.apiextensions.k8s.io/kafkarebalances.kafka.strimzi.io replaced

But still same error, besides, I could not find any sign of v1beta2 from the file 040-Crd-kafka.yaml, so how could K8S accept v1beta2 with this CRD?

[scholzj]

Please use the release-0.22.x branch and not the tag: https://github.com/strimzi/strimzi-kafka-operator/tree/release-0.22.x/helm-charts/helm3/strimzi-kafka-operator/crds

[scholzj]

(There were some changes in the file placement and the tag actually contains the 0.21 CRD on the path you used - more or less due to mistake)

[misurin]

The release-0.22.x works, all good now, thanks for answering my qns.

[scholzj]

Great. And sorry for the confusing 0.22.1 tag.