Periodic failure to reconcile: Exceeded timeout of 300000ms while waiting for Endpoints resource *-bootstrap

[blintgit]

Hello!

We are using the default operator configuration and was experiencing reconciliation failures (read timeouts) periodically on the operator.

As suggested in the following discussion: #7442
we upgraded from 0.30.0 -> 0.32.0

We are still experiencing reconciliation failures periodically (but in a different concept with the new version) which sometime lead to the restart of the operator:

2023-02-07 08:27:09 ERROR Util:167 - Reconciliation #3588(timer) Kafka(wedev/wiq): Exceeded timeout of 300000ms while waiting for Endpoints resource wiq-kafka-bootstrap in namespace wedev to be ready
2023-02-07 08:27:09 ERROR AbstractOperator:260 - Reconciliation #3588(timer) Kafka(wedev/wiq): createOrUpdate failed
io.strimzi.operator.common.operator.resource.TimeoutException: Exceeded timeout of 300000ms while waiting for Endpoints resource wiq-kafka-bootstrap in namespace wedev to be ready
        at io.strimzi.operator.common.Util$1.lambda$handle$1(Util.java:168) ~[io.strimzi.operator-common-0.33.0.jar:0.33.0]
        at io.vertx.core.impl.future.FutureImpl$3.onFailure(FutureImpl.java:153) ~[io.vertx.vertx-core-4.3.6.jar:4.3.6]
        at io.vertx.core.impl.future.FutureBase.lambda$emitFailure$1(FutureBase.java:69) ~[io.vertx.vertx-core-4.3.6.jar:4.3.6]
        at io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:174) ~[io.netty.netty-common-4.1.86.Final.jar:4.1.86.Final]
        at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:167) ~[io.netty.netty-common-4.1.86.Final.jar:4.1.86.Final]
        at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:470) ~[io.netty.netty-common-4.1.86.Final.jar:4.1.86.Final]
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:569) ~[io.netty.netty-transport-4.1.86.Final.jar:4.1.86.Final]
        at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) ~[io.netty.netty-common-4.1.86.Final.jar:4.1.86.Final]
        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[io.netty.netty-common-4.1.86.Final.jar:4.1.86.Final]
        at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[io.netty.netty-common-4.1.86.Final.jar:4.1.86.Final]
        at java.lang.Thread.run(Thread.java:833) ~[?:?]
2023-02-07 08:27:09 WARN  AbstractOperator:525 - Reconciliation #3588(timer) Kafka(wedev/wiq): Failed to reconcile
io.strimzi.operator.common.operator.resource.TimeoutException: Exceeded timeout of 300000ms while waiting for Endpoints resource wiq-kafka-bootstrap in namespace wedev to be ready
        at io.strimzi.operator.common.Util$1.lambda$handle$1(Util.java:168) ~[io.strimzi.operator-common-0.33.0.jar:0.33.0]
        at io.vertx.core.impl.future.FutureImpl$3.onFailure(FutureImpl.java:153) ~[io.vertx.vertx-core-4.3.6.jar:4.3.6]
        at io.vertx.core.impl.future.FutureBase.lambda$emitFailure$1(FutureBase.java:69) ~[io.vertx.vertx-core-4.3.6.jar:4.3.6]
        at io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:174) ~[io.netty.netty-common-4.1.86.Final.jar:4.1.86.Final]
        at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:167) ~[io.netty.netty-common-4.1.86.Final.jar:4.1.86.Final]
        at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:470) ~[io.netty.netty-common-4.1.86.Final.jar:4.1.86.Final]
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:569) ~[io.netty.netty-transport-4.1.86.Final.jar:4.1.86.Final]
        at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) ~[io.netty.netty-common-4.1.86.Final.jar:4.1.86.Final]
        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[io.netty.netty-common-4.1.86.Final.jar:4.1.86.Final]
        at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[io.netty.netty-common-4.1.86.Final.jar:4.1.86.Final]
        at java.lang.Thread.run(Thread.java:833) ~[?:?]

Anyone is experiencing the same with 0.32.0? Thanks!

[scholzj]

That means that your endpoint resources are not getting ready. You should probably check (and share if needed) how do you:

• Pods look like (state, events, readiness)
• Services look like
• Endpoints look like

[blintgit]

While checking all the resources I've noticed that one of our nodes belonging to the AKS cluster had some issues (caused by MS updating them). Only a few reources were affected, but after resolving the problem the operator does not show the errors anymore. It was a little bit sneaky :)

Thanks a lot for the hint @scholzj!

[scholzj]

Glad its resolved 👍