Failed to downgrade Strimzi Cluster operator from 0.29.0 to 0.22.1

[282ori]

Hi,
I upgrade my Strimzi Cluster Operator from 0.22.1 directly to 0.29.0 in Openshift.
I encountered some issues and I tried to downgrade to 0.22.1,
I applied the old CRDs and roles and the changed the images to the 0.22.1 version and clutser operator failes to start.

The full logs:

2022-10-25 10:45:11 WARN WatcherWebSocketListener:95 - Exec Failure: HTTP 404, Status: 404 - 404 page not found

java.net.ProtocolException: Expected HTTP 101 response but was '404 Not Found'
at okhttp3.internal.ws.RealWebSocket.checkResponse(RealWebSocket.java:219)
at okhttp3.internal.ws.RealWebSocket$2.onResponse(RealWebSocket.java:186)
at okhttp3.RealCall$AsyncCall.execute(RealCall.java:153)
at okhttp3.internal.NamedRunnable.run(NamedRunnable.java:32)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)

this error repeats itself.. and eventually:
2022-10-25 10:45:13 ERROR Main:156 - Cluster operator verticle * failed to start
io.fabric8.kubernetes.client.KubernetesClientException: 404 page not found

There are no other logs but these logs.
Can anybody help?

[scholzj]

First of all, without the full log, it is quite hard to help. Second, you will need to downgrade the CRDs to make sure the old versions are there and my guess is that is what you are missing. Not sure anyone ever tried it, so you might need to figure it out.

[scholzj]

What kind of container image are you using for the operator? And what version do you actually expect to run? Your stacktrace seems to include io.strimzi.cluster-operator-0.22.1.jar:0.22.1 as well as io.strimzi.cluster-operator-0.17.0.jar:0.17.0 => so that sounds like you are using some weird container image mixing some very old versions. That probably causes the confusing error message.

[282ori]

Those are the same logs I had when I tried to do so in other environment with wrong image, but I fixed it.
The image is of the 0.22.1 operator.
The actual logs are with the same strimzi version:

2023-01-12 12:51:30 ERROR AbstractOperator:275 - Reconciliation #0(timer) Kafka(test/my-cluster): createOrUpdated failed
io.strimzi.operator.cluster.KafkaUpgradeException: Unsupported Kafka.spec.kafka.version: 3.2.0. Supported versions are: [2.1.0, 2.5.1, 2.6.0, 2.6.1, 2.7.0, 2.3.1, 2.4.0, 2.4.1, 2.5.0, 2.1.1, 2.2.0, 2.2.1, 2.3.0]
at
io.strimzi.operator.cluster.model.KafkaVersion$Lookup.version(KafkaVersion:java:132) ~[io.strimzi.cluster-operator-0.22.1.jar:0.22.1]
at io.strimzi.operator.cluster.operator.assembly.KafkaAssemblyOperator$ReconciliationState.lambda$null$28(KafkaAssemblyOperator.java:924) ~[io.strimzi.cluster-operator-0.22.1.jar:0.22.1]
at io.vertx.core.Future.lambda$compose$3(Future.java:360) ~[io.vertx.vertx-core-3.8.5.jar:3.8.5]
at io.vertx.core.impl.FutureImpl.dispatch(FutureImpl.java:107) ~[io.vertx.vertx-core-3.8.5.jar:3.8.5]
at io.vertx.core.impl.FutureImpl.tryComplete(FutureImpl.java:152) ~[io.vertx.vertx-core-3.8.5.jar:3.8.5]
at io.vertx.core.impl.FutureImpl.complete(FutureImpl.java:113) ~[io.vertx.vertx-core-3.8.5.jar:3.8.5]
at io.vertx.core.impl.FutureImpl.handle(FutureImpl.java:178) ~[io.vertx.vertx-core-3.8.5.jar:3.8.5]
at io.vertx.core.impl.FutureImpl.handle(FutureImpl.java:21) ~[io.vertx.vertx-core-3.8.5.jar:3.8.5]
at io.vertx.core.impl.FutureImpl.dispatch(FutureImpl.java:107) ~[io.vertx.vertx-core-3.8.5.jar:3.8.5]
at io.vertx.core.impl.FutureImpl.tryComplete(FutureImpl.java:152) ~[io.vertx.vertx-core-3.8.5.jar:3.8.5]
at io.vertx.core.impl.FutureImpl.complete(FutureImpl.java:113) ~[io.vertx.vertx-core-3.8.5.jar:3.8.5]
at io.vertx.core.impl.FutureImpl.handle(FutureImpl.java:178) ~[io.vertx.vertx-core-3.8.5.jar:3.8.5]
at io.vertx.core.impl.ContextImpl.executeTask(ContextImpl.java:369) ~[io.vertx.vertx-core-3.8.5.jar:3.8.5]
at io.vertx.core.impl.EventLoopContext.lambda$executeAsync$0(EventLoopContext.java:38) ~[io.vertx.vertx-core-3.8.5.jar:3.8.5]
at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) [io.netty.netty-common-4.1.45.Final.jar:4.1.45.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) [io.netty.netty-common-4.1.45.Final.jar:4.1.45.Final]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:500) [io.netty.netty-transport-4.1.45.Final.jar:4.1.45.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) [io.netty.netty-common-4.1.45.Final.jar:4.1.45.Final]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [io.netty.netty-common-4.1.45.Final.jar:4.1.45.Final]
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [io.netty.netty-common-4.1.45.Final.jar:4.1.45.Final]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_242]

[scholzj]

Well, 0.22.1 for sure does not support Kafka 3.2.0, so while the list of supported versions is wrong, the error itself is valid at its core if you have 3.2.0 images running.

[282ori]

It is not supported indeed but I changed kafka.spec.version to 2.7.0
It's like the operator does not notice this change

[scholzj]

TBH, 0.22.1 is now what ... 2 years old? It is really hard to remember what it was doing exactly. The version is normally also carried also in annotations on the StatefulSets / Pods. Both for the Kafka version itself as well as for the inter.broker.protocol and log.message.format. These affect upgrades and downgrades because they describe what you have and not just what you want to have. So it might come from those annotations possibly? But as I said, I'm not entirely sure what was supported back then. At some point, you had to go through all the versions to upgrade / downgrade ... version by version up or down. I absolutely don't remember anymore when that changed for example. So that might be an issue as well. that 0.22.1 can downgrade from 2.7.0 to 2.6.0. But downgrade to 2.7.0 had to be done before.