question on strimzi-cluster-operator createOrUpdate failed: ideas why the cluster operator might not be able to update the kafka connect status?

[cstmgl]

Hi all,

I'm having an error, which I never had before and I'm a bit confused by it.
I'm using Strimzi in the context of Openshift with the red hat AMQ streams version 2.0.1

Anyway the error I'm experiencing is that the strimzi cluster operator seems to not realize the KafkaConnect instance gets created, even if the pods are created.
I see the status of the pods as created, with basically nothing in the logs, indicating errors and also the rest api seems to work.

sh-4.4$ curl localhost:8083
{"version":"3.0.0.redhat-00008","commit":"a92f6b92b6cb42fe","kafka_cluster_id":"xCL8hKLoSDWH5MqPEq1liA"}sh-4.4$ 

Also the status from the openshift describe for the kafka connect resource shows me:

Status:
  Conditions:
    Last Transition Time:  2022-11-29T11:08:04.478112Z
    Message:               Exceeded timeout of 300000ms while waiting for Deployment resource data-flow-kc-connect in namespace insight-miguel to be ready
    Reason:                TimeoutException
    Status:                True
    Type:                  NotReady
  Label Selector:          strimzi.io/cluster=data-flow-kc,strimzi.io/name=data-flow-kc-connect,strimzi.io/kind=KafkaConnect
  Observed Generation:     1
  Replicas:                3
  URL:                     http://data-flow-kc-connect-api.insight-miguel.svc:8083
Events:                    <none>

On the strimzi cluster operator I see a couple of errors while it waits for the secrets to be created with the cluster keys, but once the cluster keys are created it seems to proceed and generate a different kind of error (can't really understand what it is):

{
  "class": "io.strimzi.operator.common.Util$1",
  "method": "lambda$handle$1",
  "file": "Util.java",
  "line": 150,
  "exact": false,
  "location": "io.strimzi.operator-common-0.26.0.redhat-00008.jar",
  "version": "0.26.0.redhat-00008"
},
{
  "class": "io.vertx.core.impl.future.FutureImpl$3",
  "method": "onFailure",
  "file": "FutureImpl.java",
  "line": 153,
  "exact": false,
  "location": "io.vertx.vertx-core-4.1.5.redhat-00001.jar",
  "version": "4.1.5.redhat-00001"
},
{
  "class": "io.vertx.core.impl.future.FutureBase",
  "method": "lambda$emitFailure$1",
  "file": "FutureBase.java",
  "line": 69,
  "exact": false,
  "location": "io.vertx.vertx-core-4.1.5.redhat-00001.jar",
  "version": "4.1.5.redhat-00001"
},
{
  "class": "io.netty.util.concurrent.AbstractEventExecutor",
  "method": "safeExecute",
  "file": "AbstractEventExecutor.java",
  "line": 164,
  "exact": true,
  "location": "io.netty.netty-common-4.1.72.Final-redhat-00001.jar",
  "version": "4.1.72.Final-redhat-00001"
},
{
  "class": "io.netty.util.concurrent.SingleThreadEventExecutor",
  "method": "runAllTasks",
  "file": "SingleThreadEventExecutor.java",
  "line": 469,
  "exact": true,
  "location": "io.netty.netty-common-4.1.72.Final-redhat-00001.jar",
  "version": "4.1.72.Final-redhat-00001"
},
{
  "class": "io.netty.channel.nio.NioEventLoop",
  "method": "run",
  "file": "NioEventLoop.java",
  "line": 503,
  "exact": true,
  "location": "io.netty.netty-transport-4.1.72.Final-redhat-00001.jar",
  "version": "4.1.72.Final-redhat-00001"
},
{
  "class": "io.netty.util.concurrent.SingleThreadEventExecutor$4",
  "method": "run",
  "file": "SingleThreadEventExecutor.java",
  "line": 986,
  "exact": true,
  "location": "io.netty.netty-common-4.1.72.Final-redhat-00001.jar",
  "version": "4.1.72.Final-redhat-00001"
},
{
  "class": "io.netty.util.internal.ThreadExecutorMap$2",
  "method": "run",
  "file": "ThreadExecutorMap.java",
  "line": 74,
  "exact": true,
  "location": "io.netty.netty-common-4.1.72.Final-redhat-00001.jar",
  "version": "4.1.72.Final-redhat-00001"
},
{
  "class": "io.netty.util.concurrent.FastThreadLocalRunnable",
  "method": "run",
  "file": "FastThreadLocalRunnable.java",
  "line": 30,
  "exact": true,
  "location": "io.netty.netty-common-4.1.72.Final-redhat-00001.jar",
  "version": "4.1.72.Final-redhat-00001"
},
{
  "class": "java.lang.Thread",
  "method": "run",
  "file": "Thread.java",
  "line": 829,
  "exact": true,
  "location": "?",
  "version": "?"
}

What I noticed is that the service generated seems to be working, at least I can query it from both pods (the pod that the kakfa connect runs in and the pod from the strimzi cluster operator:

sh-4.4$ curl http://data-flow-kc-connect-api.insight-miguel.svc:8083
{"version":"3.0.0.redhat-00008","commit":"a92f6b92b6cb42fe","kafka_cluster_id":"xCL8hKLoSDWH5MqPEq1liA"}sh-4.4$ 
sh-4.4$ 

sh-4.4$ curl http://data-flow-kc-connect-api.insight-miguel.svc:8083
{"version":"3.0.0.redhat-00008","commit":"a92f6b92b6cb42fe","kafka_cluster_id":"xCL8hKLoSDWH5MqPEq1liA"}sh-4.4$ 
sh-4.4$ 

Any idea why the cluster operator might be having this issue? Because this seems to be preventing also the creation of the actual connectors themselves since the connect cluster never gets created (I mean the pods get created, but the connect instance no).

[scholzj]

Red Hat AMQ Streams is based on Strimzi, but it is not exactly the same. So you should probably contact Red Hat support. But without a full logs (in a readable format), it might be hard to understand what the issue is.

[cstmgl]

yeah, I'll see if this is something happening all the time and contact them if needed.

I just wanted to understand the cluster operator, how often it pings the kafka connect to see if it's up or not.
Is it a process with a limit of retries or not, because I noticed the log eventually stopped and it just got stuck in the error status.

[scholzj]

I just wanted to understand the cluster operator, how often it pings the kafka connect to see if it's up or not.
Is it a process with a limit of retries or not, because I noticed the log eventually stopped and it just got stuck in the error status.

That is hard to say without properly understanding the issue. In general, by default a reconciliation happens every 2 minutes or when the custom resource is changed. The cluster state is checked in every reconciliation.

[cstmgl]

I think the problem was with me.
I configured 3 replicas (pods) and somehow only 2 have started because I don't have more resources in my namespace.
The error is a bit weird, but for me this would make sense as the reason for this error

[scholzj]

Yeah, I think this sounds definitely possible.