Strimzi Upgrade failed at CRD update

[veerendra2]

Hello, I want to test Strimzi upgrade. Currently we are using older version of Openshift v1.11 running Strimzi 0.20.1. Soon we will deploy new AKS cluster v1.20 and then migrate current kafka cluster

Followed docs -> https://strimzi.io/docs/operators/in-development/full/deploying.html#assembly-upgrade-resources-str

On AKS cluster, deployed Strimzi v0.20.1

$ kubectl version
Client Version: version.Info{Major:"1", Minor:"22", GitVersion:"v1.22.3", GitCommit:"c92036820499fedefec0f847e2054d824aea6cd1", GitTreeState:"clean", BuildDate:"2021-10-27T18:41:28Z", GoVersion:"go1.16.9", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"20", GitVersion:"v1.20.9", GitCommit:"ed93e7ea8f833824d05578765c2ad3efbebbfebe", GitTreeState:"clean", BuildDate:"2021-10-21T03:31:08Z", GoVersion:"go1.15.14", Compiler:"gc", Platform:"linux/amd64"}

$ kubectl get pods
NAME                                         READY   STATUS    RESTARTS   AGE
sirius-dev-entity-operator-bc9d6d56d-bnfkj   3/3     Running   0          101m
sirius-dev-kafka-0                           1/1     Running   0          25m
sirius-dev-kafka-1                           1/1     Running   0          26m
sirius-dev-kafka-2                           1/1     Running   0          26m
sirius-dev-kafka-exporter-84c598b6db-8tdx2   1/1     Running   0          100m
sirius-dev-zookeeper-0                       1/1     Running   0          102m
sirius-dev-zookeeper-1                       1/1     Running   0          102m
sirius-dev-zookeeper-2                       1/1     Running   1          102m
strimzi-cluster-operator-cbc47b89b-kc52b     1/1     Running   0          10h

In order to upgrade, first I tried to upgrade to strimzi v0.22. For that I downloaded tools to convert apis and update crds

Downloaded tool to convert apis 
$ wget https://github.com/strimzi/strimzi-kafka-operator/releases/download/0.22.1/api-conversion-0.22.1.zip
cd api-conversion-0.22.1/bin

$ ./api-conversion.sh convert-resource --namespace kafka
Converting KafkaTopic resource named my-topic from namespace kafka
KafkaTopic resource named my-topic in namespace kafka has been converted
Converting KafkaUser resource named galileo from namespace kafka
KafkaUser resource named galileo in namespace kafka has been converted
Converting Kafka resource named sirius-dev from namespace kafka
Kafka resource named sirius-dev in namespace kafka has been converted

$ ./api-conversion.sh crd-upgrade --debug
Changing stored version in all Strimzi CRDs to v1beta2:
Updating v1alpha1 version of Kind KafkaBridge to be served but not stored
Updating KafkaBridge CRD
io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: PATCH at: https://kafka-upgrade-test-dns-5ab4e201.hcp.northeurope.azmk8s.io/apis/apiextensions.k8s.io/v1beta1/customresourcedefinitions/kafkabridges.kafka.strimzi.io. Message: CustomResourceDefinition.apiextensions.k8s.io "kafkabridges.kafka.strimzi.io" is invalid: spec.versions: Invalid value: []apiextensions.CustomResourceDefinitionVersion{apiextensions.CustomResourceDefinitionVersion{Name:"v1alpha1", Served:true, Storage:false, Deprecated:false, DeprecationWarning:(*string)(nil), Schema:(*apiextensions.CustomResourceValidation)(nil), Subresources:(*apiextensions.CustomResourceSubresources)(nil), AdditionalPrinterColumns:[]apiextensions.CustomResourceColumnDefinition(nil)}}: must have exactly one version marked as storage version. Received status: Status(apiVersion=v1, code=422, details=StatusDetails(causes=[StatusCause(field=spec.versions, message=Invalid value: []apiextensions.CustomResourceDefinitionVersion{apiextensions.CustomResourceDefinitionVersion{Name:"v1alpha1", Served:true, Storage:false, Deprecated:false, DeprecationWarning:(*string)(nil), Schema:(*apiextensions.CustomResourceValidation)(nil), Subresources:(*apiextensions.CustomResourceSubresources)(nil), AdditionalPrinterColumns:[]apiextensions.CustomResourceColumnDefinition(nil)}}: must have exactly one version marked as storage version, reason=FieldValueInvalid, additionalProperties={})], group=apiextensions.k8s.io, kind=CustomResourceDefinition, name=kafkabridges.kafka.strimzi.io, retryAfterSeconds=null, uid=null, additionalProperties={}), kind=Status, message=CustomResourceDefinition.apiextensions.k8s.io "kafkabridges.kafka.strimzi.io" is invalid: spec.versions: Invalid value: []apiextensions.CustomResourceDefinitionVersion{apiextensions.CustomResourceDefinitionVersion{Name:"v1alpha1", Served:true, Storage:false, Deprecated:false, DeprecationWarning:(*string)(nil), Schema:(*apiextensions.CustomResourceValidation)(nil), Subresources:(*apiextensions.CustomResourceSubresources)(nil), AdditionalPrinterColumns:[]apiextensions.CustomResourceColumnDefinition(nil)}}: must have exactly one version marked as storage version, metadata=ListMeta(_continue=null, remainingItemCount=null, resourceVersion=null, selfLink=null, additionalProperties={}), reason=Invalid, status=Failure, additionalProperties={}).
        at io.fabric8.kubernetes.client.dsl.base.OperationSupport.requestFailure(OperationSupport.java:570)
        at io.fabric8.kubernetes.client.dsl.base.OperationSupport.assertResponseCode(OperationSupport.java:509)
        at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:474)
        at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:435)
        at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handlePatch(OperationSupport.java:312)
        at io.fabric8.kubernetes.client.dsl.base.BaseOperation.handlePatch(BaseOperation.java:881)
        at io.fabric8.kubernetes.client.dsl.base.HasMetadataOperation.lambda$patch$1(HasMetadataOperation.java:121)
        at io.fabric8.kubernetes.client.dsl.base.HasMetadataOperation.patch(HasMetadataOperation.java:126)
        at io.fabric8.kubernetes.client.dsl.base.HasMetadataOperation.patch(HasMetadataOperation.java:30)
        at io.strimzi.kafka.api.conversion.cli.CrdUpgradeCommand.changeStoredVersionInSpecForKind(CrdUpgradeCommand.java:100)
        at io.strimzi.kafka.api.conversion.cli.CrdUpgradeCommand.changeStoredVersionInSpec(CrdUpgradeCommand.java:112)
        at io.strimzi.kafka.api.conversion.cli.CrdUpgradeCommand.run(CrdUpgradeCommand.java:162)
        at picocli.CommandLine.executeUserObject(CommandLine.java:1919)
        at picocli.CommandLine.access$1200(CommandLine.java:145)
        at picocli.CommandLine$RunLast.executeUserObjectOfLastSubcommandWithSameParent(CommandLine.java:2332)
        at picocli.CommandLine$RunLast.handle(CommandLine.java:2326)
        at picocli.CommandLine$RunLast.handle(CommandLine.java:2291)
        at picocli.CommandLine$AbstractParseResultHandler.execute(CommandLine.java:2159)
        at picocli.CommandLine.execute(CommandLine.java:2058)
        at io.strimzi.kafka.api.conversion.cli.EntryCommand.main(EntryCommand.java:31)

FYI I'm not using Kafka Bridge.

Any help would be appreciated

[scholzj]

TBH, I'm not sure I follow what exactly are you doing. If you start with a new AKS cluster, why do you install Strimzi 0.20 on it? Why don't you install the newer version right away? Also, the conversion tool is applied after you install Strimzi 0.22 => not before. Is that something you have done?

[veerendra2]

Thanks for the reply, no I didn't upgrade to 0.22. Seems I missed points in docs.

I want to note down the steps for upgrade process. That is an upgrade simulation, that's why I installed older version and upgrade(I know, it seems dum idea, but my team is on my a**)

[scholzj]

I want to note down the steps for upgrade process. That is an upgrade simulation, that's why I installed older version and upgrade(I know, it seems dum idea, but my team is on my a**)

Well, trying the upgrade somewhere else first is of course good. But I wonder more about the environment differences.

0.20 is the last version which supports Kubernetes 1.11 or OpenShift 3.11. So you cannot upgrade to 0.22 there. And if you want to move to AKS cluster with Kube 1.20, you will basically need to create a new cluster and for example mirror the data from the old cluster. But that also means you do not need to start with Strimzi 0.20 => you can mirror the data from Strimzi 0.20 on old cluster to 0.22 or even 0.26 on the new cluster directly, without any upgrades.