io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: GET at: https://10.100.0.1/apis/core.strimzi.io/v1beta2/strimzipodsets. Message: 404 page not found

[spr-mweber3]

Hey guys, after upgrading to v0.27.0 the operator is unable to start successfully.

io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: GET at: https://10.100.0.1/apis/core.strimzi.io/v1beta2/strimzipodsets. Message: 404 page not found

2021-12-27 08:44:12 ERROR Main:154 - Cluster Operator verticle in namespace * failed to start
io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: GET at: https://10.100.0.1/apis/core.strimzi.io/v1beta2/strimzipodsets. Message: 404 page not found
.
	at io.fabric8.kubernetes.client.dsl.base.OperationSupport.requestFailure(OperationSupport.java:697) ~[io.fabric8.kubernetes-client-5.10.1.jar:?]
	at io.fabric8.kubernetes.client.dsl.base.OperationSupport.requestFailure(OperationSupport.java:676) ~[io.fabric8.kubernetes-client-5.10.1.jar:?]
	at io.fabric8.kubernetes.client.dsl.base.OperationSupport.assertResponseCode(OperationSupport.java:629) ~[io.fabric8.kubernetes-client-5.10.1.jar:?]
	at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:566) ~[io.fabric8.kubernetes-client-5.10.1.jar:?]
	at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:527) ~[io.fabric8.kubernetes-client-5.10.1.jar:?]
	at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:510) ~[io.fabric8.kubernetes-client-5.10.1.jar:?]
	at io.fabric8.kubernetes.client.dsl.base.BaseOperation.listRequestHelper(BaseOperation.java:140) ~[io.fabric8.kubernetes-client-5.10.1.jar:?]
	at io.fabric8.kubernetes.client.dsl.base.BaseOperation.list(BaseOperation.java:422) ~[io.fabric8.kubernetes-client-5.10.1.jar:?]
	at io.fabric8.kubernetes.client.dsl.base.BaseOperation.list(BaseOperation.java:411) ~[io.fabric8.kubernetes-client-5.10.1.jar:?]
	at io.fabric8.kubernetes.client.dsl.base.BaseOperation.list(BaseOperation.java:91) ~[io.fabric8.kubernetes-client-5.10.1.jar:?]
	at io.fabric8.kubernetes.client.informers.cache.Reflector.getList(Reflector.java:53) ~[io.fabric8.kubernetes-client-5.10.1.jar:?]
	at io.fabric8.kubernetes.client.informers.cache.Reflector.listSyncAndWatch(Reflector.java:77) ~[io.fabric8.kubernetes-client-5.10.1.jar:?]
	at io.fabric8.kubernetes.client.informers.impl.DefaultSharedIndexInformer.run(DefaultSharedIndexInformer.java:146) ~[io.fabric8.kubernetes-client-5.10.1.jar:?]
	at io.fabric8.kubernetes.client.dsl.base.BaseOperation.inform(BaseOperation.java:954) ~[io.fabric8.kubernetes-client-5.10.1.jar:?]
	at io.fabric8.kubernetes.client.dsl.Informable.inform(Informable.java:52) ~[io.fabric8.kubernetes-client-5.10.1.jar:?]
	at io.strimzi.operator.common.operator.resource.AbstractResourceOperator.informer(AbstractResourceOperator.java:456) ~[io.strimzi.operator-common-0.27.0.jar:0.27.0]
	at io.strimzi.operator.cluster.operator.assembly.StrimziPodSetController.<init>(StrimziPodSetController.java:94) ~[io.strimzi.cluster-operator-0.27.0.jar:0.27.0]
	at io.strimzi.operator.cluster.ClusterOperator.start(ClusterOperator.java:99) ~[io.strimzi.cluster-operator-0.27.0.jar:0.27.0]
	at io.vertx.core.impl.DeploymentManager.lambda$doDeploy$5(DeploymentManager.java:196) ~[io.vertx.vertx-core-4.2.1.jar:4.2.1]
	at io.vertx.core.impl.AbstractContext.dispatch(AbstractContext.java:100) ~[io.vertx.vertx-core-4.2.1.jar:4.2.1]
	at io.vertx.core.impl.AbstractContext.dispatch(AbstractContext.java:63) ~[io.vertx.vertx-core-4.2.1.jar:4.2.1]
	at io.vertx.core.impl.EventLoopContext.lambda$runOnContext$0(EventLoopContext.java:38) ~[io.vertx.vertx-core-4.2.1.jar:4.2.1]
	at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) [io.netty.netty-common-4.1.71.Final.jar:4.1.71.Final]
	at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:469) [io.netty.netty-common-4.1.71.Final.jar:4.1.71.Final]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:503) [io.netty.netty-transport-4.1.71.Final.jar:4.1.71.Final]
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986) [io.netty.netty-common-4.1.71.Final.jar:4.1.71.Final]
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [io.netty.netty-common-4.1.71.Final.jar:4.1.71.Final]
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [io.netty.netty-common-4.1.71.Final.jar:4.1.71.Final]
	at java.lang.Thread.run(Thread.java:829) [?:?]

[jr200]

I saw the same error when upgrading (using the helm chart). It looks like some CRDs were updated in the 0.27.0 release.

I ran this which fixed the issue:

kubectl apply -f https://raw.githubusercontent.com/strimzi/strimzi-kafka-operator/main/packaging/helm-charts/helm3/strimzi-kafka-operator/crds/042-Crd-strimzipodset.yaml

Maybe documentation missing in the CRD section on this page: https://github.com/strimzi/strimzi-kafka-operator/tree/main/packaging/helm-charts/helm3/strimzi-kafka-operator

[scholzj]

It looks like you didn't installed the CRDs matching the 0.27.0 release. Every release has its own set of CRD and you should always install them as part of the upgrade.

[spr-mweber3]

The helm upgrade command does not upgrade the Custom Resource Definitions for Helm. Install the new CRDs manually after upgrading the Cluster Operator. You can access the CRDs from GitHub or find them in the crd subdirectory inside the Helm Chart.

Sorry for that. You're right. Will try that and I'm pretty sure it will work that way.

[GarimaBathla]

Got this link from @scholzj

for each version, you can use this link https://github.com/strimzi/strimzi-kafka-operator/releases/download/{VERSION}/strimzi-crds-{VERSION}.yaml

VERSION - 0.31.0 - I used https://github.com/strimzi/strimzi-kafka-operator/releases/download/0.31.0/strimzi-crds-0.31.0.yaml

[zencircle]

I had to add some old CRDs(previous version) and new ones