Trouble upgrading from v0.22.1 to v0.25.0

[chadxzs]

Hi there. I'm trying to go through the process of upgrading from v0.22.1 to v0.25.0. I have already run the upgrade tool to upgrade my resources to the v1beta2 CRDs in my cluster, as seen below, but when trying to import the v0.25.0 CRDs I get an error

❯ ./api-conversion-0.24.0/bin/api-conversion.sh convert-resources -a
Converting KafkaTopic resource named consumer-offsets---84e7a678d08f4bd226872e5cdd4eb527fadc1c6a from namespace build
KafkaTopic resource named consumer-offsets---84e7a678d08f4bd226872e5cdd4eb527fadc1c6a in namespace build has been converted
Converting KafkaTopic resource named events from namespace build
KafkaTopic resource named events in namespace build has been converted
Converting KafkaTopic resource named strimzi-store-topic---effb8e3e057afce1ecf67c3f5d8e4e3ff177fc55 from namespace build
KafkaTopic resource named strimzi-store-topic---effb8e3e057afce1ecf67c3f5d8e4e3ff177fc55 in namespace build has been converted
Converting KafkaTopic resource named strimzi-topic-operator-kstreams-topic-store-changelog---b75e702040b99be8a9263134de3507fc0cc4017b from namespace build
KafkaTopic resource named strimzi-topic-operator-kstreams-topic-store-changelog---b75e702040b99be8a9263134de3507fc0cc4017b in namespace build has been converted
Converting KafkaTopic resource named my-topic from namespace default
KafkaTopic resource named my-topic in namespace default has been converted
Converting KafkaUser resource named kafka-ci-branch-deployments from namespace build
KafkaUser resource named kafka-ci-branch-deployments in namespace build has been converted
Converting KafkaUser resource named kafka-ci-metrics from namespace build
KafkaUser resource named kafka-ci-metrics in namespace build has been converted
Converting KafkaUser resource named kafka-ci-slack-notifier from namespace build
KafkaUser resource named kafka-ci-slack-notifier in namespace build has been converted
Converting KafkaUser resource named kafka-ci-webhooks from namespace build
KafkaUser resource named kafka-ci-webhooks in namespace build has been converted
Converting KafkaUser resource named my-user from namespace kaphooka
KafkaUser resource named my-user in namespace kaphooka has been converted
Converting Kafka resource named ci from namespace build
Kafka resource named ci in namespace build has been converted

❯ kubectl replace -f strimzi-crds-0.25.0.yaml                       
customresourcedefinition.apiextensions.k8s.io/kafkatopics.kafka.strimzi.io replaced
customresourcedefinition.apiextensions.k8s.io/kafkausers.kafka.strimzi.io replaced
Error from server (Invalid): error when replacing "strimzi-crds-0.25.0.yaml": CustomResourceDefinition.apiextensions.k8s.io "kafkas.kafka.strimzi.io" is invalid: status.storedVersions[0]: Invalid value: "v1beta1": must appear in spec.versions
Error from server (Invalid): error when replacing "strimzi-crds-0.25.0.yaml": CustomResourceDefinition.apiextensions.k8s.io "kafkaconnects.kafka.strimzi.io" is invalid: status.storedVersions[0]: Invalid value: "v1beta1": must appear in spec.versions
Error from server (Invalid): error when replacing "strimzi-crds-0.25.0.yaml": CustomResourceDefinition.apiextensions.k8s.io "kafkamirrormakers.kafka.strimzi.io" is invalid: status.storedVersions[0]: Invalid value: "v1beta1": must appear in spec.versions
Error from server (Invalid): error when replacing "strimzi-crds-0.25.0.yaml": CustomResourceDefinition.apiextensions.k8s.io "kafkabridges.kafka.strimzi.io" is invalid: status.storedVersions[0]: Invalid value: "v1alpha1": must appear in spec.versions
Error from server (Invalid): error when replacing "strimzi-crds-0.25.0.yaml": CustomResourceDefinition.apiextensions.k8s.io "kafkaconnectors.kafka.strimzi.io" is invalid: status.storedVersions[0]: Invalid value: "v1alpha1": must appear in spec.versions
Error from server (Invalid): error when replacing "strimzi-crds-0.25.0.yaml": CustomResourceDefinition.apiextensions.k8s.io "kafkamirrormaker2s.kafka.strimzi.io" is invalid: status.storedVersions[0]: Invalid value: "v1alpha1": must appear in spec.versions
Error from server (Invalid): error when replacing "strimzi-crds-0.25.0.yaml": CustomResourceDefinition.apiextensions.k8s.io "kafkarebalances.kafka.strimzi.io" is invalid: status.storedVersions[0]: Invalid value: "v1alpha1": must appear in spec.versions

I'm looking for guidance on how to further debug and fix the error Error from server (Invalid): error when replacing "strimzi-crds-0.25.0.yaml": CustomResourceDefinition.apiextensions.k8s.io "kafkarebalances.kafka.strimzi.io" is invalid: status.storedVersions[0]: Invalid value: "v1alpha1": must appear in spec.versions

Thank you 🙏

[chadxzs]

Running this fixed it for me

❯ ./api-conversion-0.24.0/bin/api-conversion.sh crd-upgrade
Changing stored version in all Strimzi CRDs to v1beta2:
Updating KafkaBridge CRD
Updating KafkaTopic CRD
Updating KafkaConnector CRD
Updating KafkaConnectS2I CRD
Updating KafkaConnect CRD
Updating KafkaMirrorMaker2 CRD
Updating KafkaUser CRD
Updating KafkaRebalance CRD
Updating KafkaMirrorMaker CRD
Updating Kafka CRD

Updating all Strimzi CRs to be stored under v1beta2:
Updating KafkaTopic consumer-offsets---84e7a678d08f4bd226872e5cdd4eb527fadc1c6a to be stored as v1beta2
Updating KafkaTopic events to be stored as v1beta2
Updating KafkaTopic strimzi-store-topic---effb8e3e057afce1ecf67c3f5d8e4e3ff177fc55 to be stored as v1beta2
Updating KafkaTopic strimzi-topic-operator-kstreams-topic-store-changelog---b75e702040b99be8a9263134de3507fc0cc4017b to be stored as v1beta2
Updating KafkaTopic my-topic to be stored as v1beta2
Updating KafkaUser kafka-ci-branch-deployments to be stored as v1beta2
Updating KafkaUser kafka-ci-metrics to be stored as v1beta2
Updating KafkaUser kafka-ci-slack-notifier to be stored as v1beta2
Updating KafkaUser kafka-ci-webhooks to be stored as v1beta2
Updating KafkaUser my-user to be stored as v1beta2
Updating Kafka ci to be stored as v1beta2

Changing stored version in statuses of all Strimzi CRDs to v1beta2:
Updating KafkaBridge CRD
Updating KafkaTopic CRD
Updating KafkaConnector CRD
Updating KafkaConnectS2I CRD
Updating KafkaConnect CRD
Updating KafkaMirrorMaker2 CRD
Updating KafkaUser CRD
Updating KafkaRebalance CRD
Updating KafkaMirrorMaker CRD
Updating Kafka CRD

❯ kubectl replace -f strimzi-crds-0.25.0.yaml                                     
customresourcedefinition.apiextensions.k8s.io/kafkas.kafka.strimzi.io replaced
customresourcedefinition.apiextensions.k8s.io/kafkaconnects.kafka.strimzi.io replaced
customresourcedefinition.apiextensions.k8s.io/kafkatopics.kafka.strimzi.io replaced
customresourcedefinition.apiextensions.k8s.io/kafkausers.kafka.strimzi.io replaced
customresourcedefinition.apiextensions.k8s.io/kafkamirrormakers.kafka.strimzi.io replaced
customresourcedefinition.apiextensions.k8s.io/kafkabridges.kafka.strimzi.io replaced
customresourcedefinition.apiextensions.k8s.io/kafkaconnectors.kafka.strimzi.io replaced
customresourcedefinition.apiextensions.k8s.io/kafkamirrormaker2s.kafka.strimzi.io replaced
customresourcedefinition.apiextensions.k8s.io/kafkarebalances.kafka.strimzi.io replaced

[scholzj]

Yeah, exactly - the crd-upgrade step changes the stored versions in the status of the custom resource definition and only that allows you to overwrite it.