API conversion tool not working properly #6325

hausanstreicha · 2022-02-08T13:17:52Z

hausanstreicha
Feb 8, 2022

Describe the bug
API conversion tool not working properly: custom resources and CRDs are not getting converted. Therefore I'm not able to update strimzi and kafka to 0.27.1 and 3.1.

To Reproduce
Steps to reproduce the behavior:

Strimzi 0.20.1 update via helm

NAME    NAMESPACE       REVISION        UPDATED                                 STATUS          CHART                           APP VERSION
strimzi kafka           4               2022-02-08 08:16:17.422124082 +0000 UTC deployed        strimzi-kafka-operator-0.22.1   0.22.1

download and extract api conversion tool

wget https://github.com/strimzi/strimzi-kafka-operator/releases/download/0.24.0/api-conversion-0.24.0.tar.gz
tar -xf api-conversion-0.24.0.tar.gz

Run api conversion tool

bin/api-conversion.sh convert-resource --all-namespaces

Converting KafkaTopic resource named xsuitearchivefilesystem from namespace kafka
KafkaTopic resource named xsuitearchivefilesystem in namespace kafka has been converted
Converting KafkaTopic resource named xsuiteconfig from namespace kafka
KafkaTopic resource named xsuiteconfig in namespace kafka has been converted
Converting Kafka resource named kafka-cluster from namespace kafka
Kafka resource named kafka-cluster in namespace kafka has been converted

output is successful, but api versions aren't adjusted

kubectl get crd kafkas.kafka.strimzi.io -o jsonpath="{.spec.versions[*].name}"
v1beta1 v1alpha1b

strimzi logs:

strimzi-cluster-operator-54999dfdd6-zmtkh strimzi-cluster-operator 2022-02-08 12:56:38 INFO  AbstractOperator:493 - Reconciliation #145(timer) Kafka(kafka/kafka-cluster): reconciled
strimzi-cluster-operator-54999dfdd6-zmtkh strimzi-cluster-operator 2022-02-08 12:58:35 INFO  ClusterOperator:125 - Triggering periodic reconciliation for namespace kafka...
strimzi-cluster-operator-54999dfdd6-zmtkh strimzi-cluster-operator 2022-02-08 12:58:35 INFO  ClusterOperator:125 - Triggering periodic reconciliation for namespace xsuite...
strimzi-cluster-operator-54999dfdd6-zmtkh strimzi-cluster-operator 2022-02-08 12:58:35 WARN  AbstractOperator:97 - Kafka resource kafka-cluster in namespace kafka: In API version kafka.strimzi.io/v1beta1 the metrics property at path spec.kafka.metrics has been deprecated, and should now be configured using spec.kafka.metricsConfig. This property is removed in API version v1beta2.
strimzi-cluster-operator-54999dfdd6-zmtkh strimzi-cluster-operator 2022-02-08 12:58:35 WARN  AbstractOperator:97 - Kafka resource kafka-cluster in namespace kafka: In API version kafka.strimzi.io/v1beta1 the metrics property at path spec.zookeeper.metrics has been deprecated, and should now be configured using spec.zookeeper.metricsConfig. This property is removed in API version v1beta2.
strimzi-cluster-operator-54999dfdd6-zmtkh strimzi-cluster-operator 2022-02-08 12:58:35 WARN  AbstractOperator:97 - Kafka resource kafka-cluster in namespace kafka: In API version kafka.strimzi.io/v1beta1 the metrics property at path spec.kafka.metrics has been deprecated, and should now be configured using spec.kafka.metricsConfig. This property is removed in API version v1beta2.
strimzi-cluster-operator-54999dfdd6-zmtkh strimzi-cluster-operator 2022-02-08 12:58:35 WARN  AbstractOperator:97 - Kafka resource kafka-cluster in namespace kafka: In API version kafka.strimzi.io/v1beta1 the metrics property at path spec.zookeeper.metrics has been deprecated, and should now be configured using spec.zookeeper.metricsConfig. This property is removed in API version v1beta2.
strimzi-cluster-operator-54999dfdd6-zmtkh strimzi-cluster-operator 2022-02-08 12:58:35 INFO  AbstractOperator:254 - Reconciliation #146(timer) Kafka(kafka/kafka-cluster): Kafka kafka-cluster will be checked for creation or modification
strimzi-cluster-operator-54999dfdd6-zmtkh strimzi-cluster-operator 2022-02-08 12:58:37 INFO  AbstractOperator:493 - Reconciliation #146(timer) Kafka(kafka/kafka-cluster): reconciled
strimzi-cluster-operator-54999dfdd6-zmtkh strimzi-cluster-operator 2022-02-08 13:00:35 INFO  ClusterOperator:125 - Triggering periodic reconciliation for namespace kafka...
strimzi-cluster-operator-54999dfdd6-zmtkh strimzi-cluster-operator 2022-02-08 13:00:35 INFO  ClusterOperator:125 - Triggering periodic reconciliation for namespace xsuite...
strimzi-cluster-operator-54999dfdd6-zmtkh strimzi-cluster-operator 2022-02-08 13:00:35 WARN  AbstractOperator:97 - Kafka resource kafka-cluster in namespace kafka: In API version kafka.strimzi.io/v1beta1 the metrics property at path spec.kafka.metrics has been deprecated, and should now be configured using spec.kafka.metricsConfig. This property is removed in API version v1beta2.
strimzi-cluster-operator-54999dfdd6-zmtkh strimzi-cluster-operator 2022-02-08 13:00:35 WARN  AbstractOperator:97 - Kafka resource kafka-cluster in namespace kafka: In API version kafka.strimzi.io/v1beta1 the metrics property at path spec.zookeeper.metrics has been deprecated, and should now be configured using spec.zookeeper.metricsConfig. This property is removed in API version v1beta2.
strimzi-cluster-operator-54999dfdd6-zmtkh strimzi-cluster-operator 2022-02-08 13:00:35 WARN  AbstractOperator:97 - Kafka resource kafka-cluster in namespace kafka: In API version kafka.strimzi.io/v1beta1 the metrics property at path spec.kafka.metrics has been deprecated, and should now be configured using spec.kafka.metricsConfig. This property is removed in API version v1beta2.
strimzi-cluster-operator-54999dfdd6-zmtkh strimzi-cluster-operator 2022-02-08 13:00:35 WARN  AbstractOperator:97 - Kafka resource kafka-cluster in namespace kafka: In API version kafka.strimzi.io/v1beta1 the metrics property at path spec.zookeeper.metrics has been deprecated, and should now be configured using spec.zookeeper.metricsConfig. This property is removed in API version v1beta2.
strimzi-cluster-operator-54999dfdd6-zmtkh strimzi-cluster-operator 2022-02-08 13:00:35 INFO  AbstractOperator:254 - Reconciliation #147(timer) Kafka(kafka/kafka-cluster): Kafka kafka-cluster will be checked for creation or modification
strimzi-cluster-operator-54999dfdd6-zmtkh strimzi-cluster-operator 2022-02-08 13:00:38 INFO  AbstractOperator:493 - Reconciliation #147(timer) Kafka(kafka/kafka-cluster): reconciled

since step 4 didn't work properly, the crd-upgrade fails, too:

bin/api-conversion.sh crd-upgrade
Changing stored version in all Strimzi CRDs to v1beta2:
Updating Kafka CRD
io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: PATCH at: https://10.83.18.60:6443/apis/apiextensions.k8s.io/v1beta1/customresourcedefinitions/kafkas.kafka.strimzi.io. Message: CustomResourceDefinition.apiextensions.k8s.io "kafkas.kafka.strimzi.io" is invalid: spec.versions: Invalid value: []apiextensions.CustomResourceDefinitionVersion{apiextensions.CustomResourceDefinitionVersion{Name:"v1beta1", Served:true, Storage:false, Deprecated:false, DeprecationWarning:(*string)(nil), Schema:(*apiextensions.CustomResourceValidation)(nil), Subresources:(*apiextensions.CustomResourceSubresources)(nil), AdditionalPrinterColumns:[]apiextensions.CustomResourceColumnDefinition(nil)}, apiextensions.CustomResourceDefinitionVersion{Name:"v1alpha1", Served:true, Storage:false, Deprecated:false, DeprecationWarning:(*string)(nil), Schema:(*apiextensions.CustomResourceValidation)(nil), Subresources:(*apiextensions.CustomResourceSubresources)(nil), AdditionalPrinterColumns:[]apiextensions.CustomResourceColumnDefinition(nil)}}: must have exactly one version marked as storage version. Received status: Status(apiVersion=v1, code=422, details=StatusDetails(causes=[StatusCause(field=spec.versions, message=Invalid value: []apiextensions.CustomResourceDefinitionVersion{apiextensions.CustomResourceDefinitionVersion{Name:"v1beta1", Served:true, Storage:false, Deprecated:false, DeprecationWarning:(*string)(nil), Schema:(*apiextensions.CustomResourceValidation)(nil), Subresources:(*apiextensions.CustomResourceSubresources)(nil), AdditionalPrinterColumns:[]apiextensions.CustomResourceColumnDefinition(nil)}, apiextensions.CustomResourceDefinitionVersion{Name:"v1alpha1", Served:true, Storage:false, Deprecated:false, DeprecationWarning:(*string)(nil), Schema:(*apiextensions.CustomResourceValidation)(nil), Subresources:(*apiextensions.CustomResourceSubresources)(nil), AdditionalPrinterColumns:[]apiextensions.CustomResourceColumnDefinition(nil)}}: must have exactly one version marked as storage version, reason=FieldValueInvalid, additionalProperties={})], group=apiextensions.k8s.io, kind=CustomResourceDefinition, name=kafkas.kafka.strimzi.io, retryAfterSeconds=null, uid=null, additionalProperties={}), kind=Status, message=CustomResourceDefinition.apiextensions.k8s.io "kafkas.kafka.strimzi.io" is invalid: spec.versions: Invalid value: []apiextensions.CustomResourceDefinitionVersion{apiextensions.CustomResourceDefinitionVersion{Name:"v1beta1", Served:true, Storage:false, Deprecated:false, DeprecationWarning:(*string)(nil), Schema:(*apiextensions.CustomResourceValidation)(nil), Subresources:(*apiextensions.CustomResourceSubresources)(nil), AdditionalPrinterColumns:[]apiextensions.CustomResourceColumnDefinition(nil)}, apiextensions.CustomResourceDefinitionVersion{Name:"v1alpha1", Served:true, Storage:false, Deprecated:false, DeprecationWarning:(*string)(nil), Schema:(*apiextensions.CustomResourceValidation)(nil), Subresources:(*apiextensions.CustomResourceSubresources)(nil), AdditionalPrinterColumns:[]apiextensions.CustomResourceColumnDefinition(nil)}}: must have exactly one version marked as storage version, metadata=ListMeta(_continue=null, remainingItemCount=null, resourceVersion=null, selfLink=null, additionalProperties={}), reason=Invalid, status=Failure, additionalProperties={}).
        at io.fabric8.kubernetes.client.dsl.base.OperationSupport.requestFailure(OperationSupport.java:639)
        at io.fabric8.kubernetes.client.dsl.base.OperationSupport.assertResponseCode(OperationSupport.java:578)
        at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:543)
        at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:504)
        at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handlePatch(OperationSupport.java:400)
        at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handlePatch(OperationSupport.java:361)
        at io.fabric8.kubernetes.client.dsl.base.BaseOperation.handlePatch(BaseOperation.java:903)
        at io.fabric8.kubernetes.client.dsl.base.HasMetadataOperation.lambda$patch$1(HasMetadataOperation.java:176)
        at io.fabric8.kubernetes.client.dsl.base.HasMetadataOperation.patch(HasMetadataOperation.java:181)
        at io.fabric8.kubernetes.client.dsl.base.HasMetadataOperation.patch(HasMetadataOperation.java:191)
        at io.fabric8.kubernetes.client.dsl.base.HasMetadataOperation.patch(HasMetadataOperation.java:38)
        at io.fabric8.kubernetes.client.dsl.Patchable.patch(Patchable.java:35)
        at io.strimzi.kafka.api.conversion.cli.CrdUpgradeCommand.changeStoredVersionInSpecForKind(CrdUpgradeCommand.java:100)
        at io.strimzi.kafka.api.conversion.cli.CrdUpgradeCommand.changeStoredVersionInSpec(CrdUpgradeCommand.java:112)
        at io.strimzi.kafka.api.conversion.cli.CrdUpgradeCommand.run(CrdUpgradeCommand.java:162)
        at picocli.CommandLine.executeUserObject(CommandLine.java:1919)
        at picocli.CommandLine.access$1200(CommandLine.java:145)
        at picocli.CommandLine$RunLast.executeUserObjectOfLastSubcommandWithSameParent(CommandLine.java:2332)
        at picocli.CommandLine$RunLast.handle(CommandLine.java:2326)
        at picocli.CommandLine$RunLast.handle(CommandLine.java:2291)
        at picocli.CommandLine$AbstractParseResultHandler.execute(CommandLine.java:2159)
        at picocli.CommandLine.execute(CommandLine.java:2058)
        at io.strimzi.kafka.api.conversion.cli.EntryCommand.main(EntryCommand.java:31)

Expected behavior
the custom resources and CRDs should be converted properly.

Environment (please complete the following information):

Strimzi version: 0.22.1
Installation method: Helm chart
Kubernetes cluster: Kubernetes 1.20.7
Infrastructure: Bare Metal K8s

Additional Information:
Trying to replace the CRD with the CRD from 0.27.1 (https://raw.githubusercontent.com/strimzi/strimzi-kafka-operator/0.27.1/helm-charts/helm3/strimzi-kafka-operator/crds/040-Crd-kafka.yaml) also fails, since v1beta1 doesn't appear in the spec.versions:

kubectl replace -f /tmp/040-Crd-kafka.yaml
The CustomResourceDefinition "kafkas.kafka.strimzi.io" is invalid: status.storedVersions[0]: Invalid value: "v1beta1": must appear in spec.versions

You can find my currently installed kafkas.kafka.strimzi.io CRD in the appendix.
kafkas.kafka.strimzi.io.txt

Thanks for your help in advance.

scholzj · 2022-02-08T14:17:15Z

scholzj
Feb 8, 2022
Maintainer

The step 4 suggests that you do not have th CRDs from 0.22.1 installed. If you would, it would show that you have versions v1slpha1, v1beta1andv1beta2`. Bu you seem to be missing the new version. That is most likely also the reason why the step 5 failed. From some version on (Helm version I mean => this is a Helm feature), Helm does not update the CRDs anymore, so you might need to update them manually everytime you update strimzi.

0 replies

hausanstreicha · 2022-02-08T14:35:15Z

hausanstreicha
Feb 8, 2022
Author

Thanks for you quick reply, but isn't that the point of the api conversion tool:

convert-resource:

This procedure describes how to use the API conversion tool to convert Strimzi custom resources directly in the Kubernetes cluster into a format applicable to v1beta2

crd-upgrade:

This procedure describes how to use the API conversion tool to convert the CRDs that define the schemas used to instantiate and manage Strimzi-specific resources in a format applicable to v1beta2

And like I wrote, I can't replace or apply the CRDs manually, too:

kubectl replace -f /tmp/040-Crd-kafka.yaml
The CustomResourceDefinition "kafkas.kafka.strimzi.io" is invalid: status.storedVersions[0]: Invalid value: "v1beta1": must appear in spec.versions

As far as I understand this is the recommended way in the official documentation:

1. Upgrade Strimzi to 0.22.0

2. Convert the custom resources to v1beta2

3. Upgrade Strimzi to 0.23.0 or newer

Strimzi provides an API conversion tool with its release artifacts.

You can download its ZIP or TAR.GZ file from [GitHub](https://github.com/strimzi/strimzi-kafka-operator/releases). To use the tool, extract it and use the scripts in the bin directory.

From its CLI, you can then use the tool to convert the format of your custom resources to v1beta2 in one of two ways:

Converting custom resources configuration files using the API conversion tool

Converting custom resources directly using the API conversion tool

I followed the second way. Or did I do something wrong? Or more specifically what is the recommended way to update the CRDs?

Thanks again for your help in advance.

2 replies

scholzj Feb 8, 2022
Maintainer

Well, I do not know in detail how exactly does your system look like. But the suggestion from the previous comment means that you didn't completed the Step 1 Upgrade Strimzi to 0.22.0. This is completely only once you install the CRDs from Strimzi 0.22.0 (or 0.22.1 in your case which would be fine as well).

From my point of the descriptions are correct:

This procedure describes how to use the API conversion tool to convert Strimzi custom resources directly in the Kubernetes cluster into a format applicable to v1beta2 => it converts the custom resources, but you need to have the right Custom Resource Definitions installed in the first place
This procedure describes how to use the API conversion tool to convert the CRDs that define the schemas used to instantiate and manage Strimzi-specific resources in a format applicable to v1beta2 => I guess this is a bit generic. But it really means that is just changes the used versions of the CRD. But it does not add the new versions into the CRDs. They should be present there already when you upgrade to 0.22.

scholzj Feb 8, 2022
Maintainer

And like I wrote, I can't replace or apply the CRDs manually, too:
kubectl replace -f /tmp/040-Crd-kafka.yaml
The CustomResourceDefinition "kafkas.kafka.strimzi.io" is invalid: status.storedVersions[0]: Invalid value: "v1beta1": must appear in spec.versions

This is when you try to update them manually to 0.27 CRDs. But you first need to have the 0.22 CRDs installed, then do the conversion and only then you can install the 0.27 CRDs (you should do it together with the operator upgrade in any case, not just install them while keeping the 0.22 operator).

hausanstreicha · 2022-02-09T08:14:18Z

hausanstreicha
Feb 9, 2022
Author

Thanks again for your quick reply, but sadly I'm still lost here.

If I run the convert-resource in debug mode it even tells me, that the api version was changed to kafka.strimzi.io/v1beta2, but in reality this didn't happen:

bin/api-conversion.sh convert-resource --all-namespaces -d
Converting Kafka resource named kafka-cluster from namespace kafka
Kafka resource named kafka-cluster in namespace kafka has been converted
Converted custom resource:
---
apiVersion: "kafka.strimzi.io/v1beta2"
kind: "Kafka"
metadata:
  annotations:
    prometheus.io/path: "/metrics"
    prometheus.io/port: "9404"
    prometheus.io/scrape: "true"
  creationTimestamp: "2022-02-04T12:17:54Z"
  generation: 5
  labels:
    app: "kafka-cluster"

Anyways I installed the CRDs from release 0.22.1 as you said and then tried to run the convert-resource and crd-upgrade scripts again. But nothing changed and I'm getting the same errors as before, which makes sense to me, because all CRDs from release 0.22.1 only contain v1beta1 versions. Therefore I'm not able run the scripts and adjust the api versions properly.

And I think this is the main problem here. This is why I also tried installing the CRDs from release 0.27.1 in the first place. The problem for most of the 0.27.1 CRDs is, that they only contain the v1beta2 versions. Therefore I'm not able to replace the CRDs. Some 0.27.1 CRDs contain all api versions (v1alpha1, v1beta1 and v1beta2). This is for example the case for the kafkatopics CRD. In this case I was able to replace the CRD and run the convert-resource and crd-upgrade scripts successfully.

This is why I'm a bit lost here, because I don't know how I should proceed with the update, without deleting all relevant CRDs and installing them new, which isn't a good solution for me. Is there a possibility that the other CRDs will get adjusted, so that they contain all api versions like the kafkatopics CRD? Or is there something else I could try?

Thanks again for your help in advance!

1 reply

hausanstreicha Feb 9, 2022
Author

I found the problem in another discussion: #5482

I was using the CRDs from the tag 0.22.1 (they are not correct there) and not from the branch release-0.22.x. I didn't know that they were different.

It is working now. Thanks for your help.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Strimzi

API conversion tool not working properly #6325

Uh oh!

{{title}}

Uh oh!

Replies: 3 comments 3 replies

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Select a reply

Uh oh!

Strimzi

API conversion tool not working properly #6325

Uh oh!

hausanstreicha Feb 8, 2022

Replies: 3 comments · 3 replies

Uh oh!

scholzj Feb 8, 2022 Maintainer

Uh oh!

hausanstreicha Feb 8, 2022 Author

Uh oh!

scholzj Feb 8, 2022 Maintainer

Uh oh!

scholzj Feb 8, 2022 Maintainer

Uh oh!

hausanstreicha Feb 9, 2022 Author

Uh oh!

hausanstreicha Feb 9, 2022 Author

hausanstreicha
Feb 8, 2022

Replies: 3 comments 3 replies

scholzj
Feb 8, 2022
Maintainer

hausanstreicha
Feb 8, 2022
Author

scholzj Feb 8, 2022
Maintainer

scholzj Feb 8, 2022
Maintainer

hausanstreicha
Feb 9, 2022
Author

hausanstreicha Feb 9, 2022
Author