error upgrading Kafka CRDs t0 0.29 - Invalid value: "v1beta": must appear in spec.versions

[282ori]

Hi,
I'm trying to upgrade the Kafka CRD's from 0.22.1 to 0.29.
when I run the oc replace command there's an exception:

CustomResourceDefinition.apiextensions.k8s.io "kafkas.kafka.strimzi.io" is invalid: status.storedVersions[0]:
Invalid value: "v1beta1": must appear in spec.versions

I receive this error when I try to replace the CRDs: kafka, kafka connect, kafkaconnector, kafkamirrormaker,kafkarebalance etc.
The only CRDs I succeeded replacing are KafkaTopic and KafkaUser.
for some of the CRDs instead of 'invalid value: v1beta1' I see 'invalid value: v1alpha1'

Does anyone know what could be the problem?
Thanks

[scholzj]

Please follow the CRD upgrade procedure. It is linked from the release page and from the docs.

[scholzj]

It is pointless to 'kubectl replace' before the script would run successfully

kubectl apply is known to cause issues during patching of complex resources. The patch is sometimes simply wrong. kubectl replace is the command which avoid the risk of bad patch and makes sure the CRD is as it is in the YAML file. So it is not pointless. Given your current state, you should download the release ZIP, use kubectl replace to make sure you have the right CRDs from 0.22.0 or 0.22.1.

I also saw that there are serveral versions of the api-conversion tool, does it matter which version of it I am running?

The last version - from 0.24 - has IIRC some bugfix for converting CR with some special configuration. It was some niche thing IIRC, but you might be best of to use that version.

[javadnasrolahi]

Hi
I think we have the same problem.
we upgraded strimzi from version 0.19.0 to 0.22.1 directly. then we changed all Kafka resources from v1beta1 to v1beta2 as it was said in the document without any problem. (we didn't use API conversion tool)

then we went to upgrade strimzi from version 0.22.1 to version 0.25.0. first, we upgrade the operator with helm upgrade without any problem, then we wanted to upgrade CRDs so we run kubectl replace -f strimzi-kafka-operator-0.25.0/crds/ but we got this err:

customresourcedefinition.apiextensions.k8s.io/kafkatopics.kafka.strimzi.io replaced
customresourcedefinition.apiextensions.k8s.io/kafkausers.kafka.strimzi.io replaced
Error from server (Invalid): error when replacing "strimzi-kafka-operator-0.25.0/crds/040-Crd-kafka.yaml": CustomResourceDefinition.apiextensions.k8s.io "kafkas.kafka.strimzi.io" is invalid: status.storedVersions[0]: Invalid value: "v1beta1": must appear in spec.versions
Error from server (Invalid): error when replacing "strimzi-kafka-operator-0.25.0/crds/041-Crd-kafkaconnect.yaml": CustomResourceDefinition.apiextensions.k8s.io "kafkaconnects.kafka.strimzi.io" is invalid: status.storedVersions[0]: Invalid value: "v1beta1": must appear in spec.versions
Error from server (Invalid): error when replacing "strimzi-kafka-operator-0.25.0/crds/045-Crd-kafkamirrormaker.yaml": CustomResourceDefinition.apiextensions.k8s.io "kafkamirrormakers.kafka.strimzi.io" is invalid: status.storedVersions[0]: Invalid value: "v1beta1": must appear in spec.versions
Error from server (Invalid): error when replacing "strimzi-kafka-operator-0.25.0/crds/046-Crd-kafkabridge.yaml": CustomResourceDefinition.apiextensions.k8s.io "kafkabridges.kafka.strimzi.io" is invalid: status.storedVersions[0]: Invalid value: "v1alpha1": must appear in spec.versions
Error from server (Invalid): error when replacing "strimzi-kafka-operator-0.25.0/crds/047-Crd-kafkaconnector.yaml": CustomResourceDefinition.apiextensions.k8s.io "kafkaconnectors.kafka.strimzi.io" is invalid: status.storedVersions[0]: Invalid value: "v1alpha1": must appear in spec.versions
Error from server (Invalid): error when replacing "strimzi-kafka-operator-0.25.0/crds/048-Crd-kafkamirrormaker2.yaml": CustomResourceDefinition.apiextensions.k8s.io "kafkamirrormaker2s.kafka.strimzi.io" is invalid: status.storedVersions[0]: Invalid value: "v1alpha1": must appear in spec.versions
Error from server (Invalid): error when replacing "strimzi-kafka-operator-0.25.0/crds/049-Crd-kafkarebalance.yaml": CustomResourceDefinition.apiextensions.k8s.io "kafkarebalances.kafka.strimzi.io" is invalid: status.storedVersions[0]: Invalid value: "v1alpha1": must appear in spec.versions

[scholzj]

Have you done the conversion using the conversion tool as described in the docs? It does not seem so from the error message.

[javadnasrolahi]

I think we did (maybe we didn't do a part correctly)
we changed all v1beta1 resources. here is our manifest:

---
apiVersion: kafka.strimzi.io/v1beta2
kind: Kafka
metadata:
  name: kafka
  labels:
    app: kafka
spec:
  kafka:
    version: 2.7.0
    replicas: 5
    template:
    jvmOptions:
      -Xms: 7000m
      -Xmx: 7000m
      javaSystemProperties:
        - name: log4j2.formatMsgNoLookups
          value: "true"
    listeners:
      - name: plain
        port: 9092
        type: internal
        tls: false
      - name: tls
        port: 9093
        type: internal
        tls: true
    readinessProbe:
      initialDelaySeconds: 15
      timeoutSeconds: 5
    livenessProbe:
      initialDelaySeconds: 15
      timeoutSeconds: 5
    config:
      log.message.format.version: "2.7"
    storage:
      type: persistent-claim
      size: 23Gi
      deleteClaim: false
      class: "lvm-ext4"
    metricsConfig:
      type: jmxPrometheusExporter
      valueFrom:
        configMapKeyRef:
          name: kafka-metrics
          key: kafka-metrics-config.yaml
  zookeeper:
    replicas: 3
    jvmOptions:
      -Xms: 2024m
      -Xmx: 2024m
      javaSystemProperties:
        - name: log4j2.formatMsgNoLookups
          value: "true"
    template:
    readinessProbe:
      initialDelaySeconds: 15
      timeoutSeconds: 5
    livenessProbe:
      initialDelaySeconds: 15
      timeoutSeconds: 5
    storage:
      type: persistent-claim
      size: 15Gi
      deleteClaim: false
      class: "lvm-ext4"
    metricsConfig:
      type: jmxPrometheusExporter
      valueFrom:
        configMapKeyRef:
          name: zookeeper-metrics
          key: zookeeper-metrics-config.yaml

  kafkaExporter:
    #image: "danielqsj/kafka-exporter:v1.3.1"
    topicRegex: ".*."
    groupRegex: ".*."
    resources:
      requests:
        cpu: 200m
        memory: 64Mi
      limits:
        cpu: 500m
        memory: 128Mi
    logging: "warn"
    enableSaramaLogging: true
  entityOperator:
    topicOperator:
      jvmOptions:
        javaSystemProperties:
          - name: log4j2.formatMsgNoLookups
            value: "true"
    userOperator:
      jvmOptions:
        javaSystemProperties:
          - name: log4j2.formatMsgNoLookups
            value: "true"

[scholzj]

But that is not all what the conversion tool does. It also changes the CRDs themself and their status. So you should use it.