Strimzi operator cannot start

[woot3d]

I'm using strimzi (0.19.0) ,it was working correctly without any issues ,and somehow now it wont start ,we did not had any change on the system ,that's why it sounds pretty werid ,here is what appears in logs :

WatchConnectionManager:202 - Exec Failure: HTTP 404, Status: 404 - 404 page not found
t java.lang.Thread.run(Thread.java:834) [?:?]
Exception in thread "OkHttp Dispatcher" java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask@5d29f30[Not completed, task = java.util.concurrent.Executors$RunnableAdapter@57d8532d[Wrapped task = io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager$2@100d9a7e]] rejected from java.util.concurrent.ScheduledThreadPoolExecutor@33759be4[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 0]
	at java.base/java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2055)
	at java.base/java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:825)
	at java.base/java.util.concurrent.ScheduledThreadPoolExecutor.delayedExecute(ScheduledThreadPoolExecutor.java:340)
	at java.base/java.util.concurrent.ScheduledThreadPoolExecutor.schedule(ScheduledThreadPoolExecutor.java:562)
	at java.base/java.util.concurrent.ScheduledThreadPoolExecutor.submit(ScheduledThreadPoolExecutor.java:715)
	at java.base/java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:714)
	at io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager.scheduleReconnect(WatchConnectionManager.java:318)
	at io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager.access$800(WatchConnectionManager.java:50)
	at io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager$1.onFailure(WatchConnectionManager.java:221)
	at okhttp3.internal.ws.RealWebSocket.failWebSocket(RealWebSocket.java:571)
	at okhttp3.internal.ws.RealWebSocket$2.onResponse(RealWebSocket.java:198)
	at okhttp3.RealCall$AsyncCall.execute(RealCall.java:203)
	at okhttp3.internal.NamedRunnable.run(NamedRunnable.java:32)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.base/java.lang.Thread.run(Thread.java:834)
2021-03-13 04:16:12 ERROR Main:162 - Cluster Operator verticle in namespace failed to start
io.fabric8.kubernetes.client.KubernetesClientException: 404 page not found

	at io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager$1.onFailure(WatchConnectionManager.java:206) ~[io.fabric8.kubernetes-client-4.6.4.jar:?]
	at okhttp3.internal.ws.RealWebSocket.failWebSocket(RealWebSocket.java:571) ~[com.squareup.okhttp3.okhttp-3.12.6.jar:?]
	at okhttp3.internal.ws.RealWebSocket$2.onResponse(RealWebSocket.java:198) ~[com.squareup.okhttp3.okhttp-3.12.6.jar:?]
	at okhttp3.RealCall$AsyncCall.execute(RealCall.java:203) ~[com.squareup.okhttp3.okhttp-3.12.6.jar:?]
	at okhttp3.internal.NamedRunnable.run(NamedRunnable.java:32) ~[com.squareup.okhttp3.okhttp-3.12.6.jar:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
	at java.lang.Thread.run(Thread.java:834) [?:?]
	Suppressed: java.lang.Throwable: waiting here
		at io.fabric8.kubernetes.client.utils.Utils.waitUntilReady(Utils.java:135) ~[io.fabric8.kubernetes-client-4.6.4.jar:?]
		at io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager.waitUntilReady(WatchConnectionManager.java:354) ~[io.fabric8.kubernetes-client-4.6.4.jar:?]
		at io.fabric8.kubernetes.client.dsl.base.BaseOperation.watch(BaseOperation.java:759) ~[io.fabric8.kubernetes-client-4.6.4.jar:?]
		at io.fabric8.kubernetes.client.dsl.base.BaseOperation.watch(BaseOperation.java:738) ~[io.fabric8.kubernetes-client-4.6.4.jar:?]
		at io.fabric8.kubernetes.client.dsl.base.BaseOperation.watch(BaseOperation.java:69) ~[io.fabric8.kubernetes-client-4.6.4.jar:?]
		at io.strimzi.operator.common.operator.resource.AbstractWatchableResourceOperator.watchInNamespace(AbstractWatchableResourceOperator.java:46) ~[io.strimzi.operator-common-0.19.0.jar:0.19.0]
		at io.strimzi.operator.common.operator.resource.AbstractWatchableResourceOperator.watch(AbstractWatchableResourceOperator.java:53) ~[io.strimzi.operator-common-0.19.0.jar:0.19.0]
		at io.strimzi.operator.cluster.operator.assembly.KafkaRebalanceAssemblyOperator.lambda$createRebalanceWatch$0(KafkaRebalanceAssemblyOperator.java:183) ~[io.strimzi.cluster-operator-0.19.0.jar:0.19.0]
		at io.strimzi.operator.common.Util.lambda$async$0(Util.java:49) ~[io.strimzi.operator-common-0.19.0.jar:0.19.0]
		at io.vertx.core.impl.ContextImpl.lambda$executeBlocking$2(ContextImpl.java:313) ~[io.vertx.vertx-core-3.9.1.jar:3.9.1]
		at io.vertx.core.impl.TaskQueue.run(TaskQueue.java:76) ~[io.vertx.vertx-core-3.9.1.jar:3.9.1]
		at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
		at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
		at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [io.netty.netty-common-4.1.50.Final.jar:4.1.50.Final]
		at java.lang.Thread.run(Thread.java:834) [?:?]

any help is greatly appreciated .

[scholzj]

This looks like someone / something deleted the Strimzi CRDs. So you would need to install them again.

[woot3d]

Now it makes more sense for me , thanks a lot @scholzj really appreciate your help

[woot3d]

My last question just to confirm my upcoming steps ,if i will deploy a CR and maybe this CR will trigger an operator ,and that operator deployment will fail for some raison , my understanding is to fix that is by troubleshooting the CR that caused that operator to start as i cannot in any way modify or adjust or fix that new operator settings ! and focusing on fixing that new operator will not work in any way as that will be always overridden by the parameters already defined in the CR .

so the troubleshooting/fix will be always on the CR that triggered that operator to start ! am i right @scholzj

[woot3d]

Ok, so other theory ... maybe your operator is watching multiple namespaces:

Cluster Operator verticle in namespace arcchfdev failed to start

Could it be that this namespace arcchfdev does not exist anymore?

@scholzj Team asked me to remove some details form the logs output (like the namespace name ) which i did , kindly can we remove it from here too !

[scholzj]

Yes, to fix something about the operands (i.e. everything deployed based on a custom resource you created - including the entity operator) is usually done by adding or removing things in the custom resource it self.

I edited out the original namespace name.

[woot3d]

@scholzj much appreciate all your help