KafkaConnect Pending State/CrashLoopBackOffs #8608

imrohankataria · 2023-06-04T22:11:24Z

imrohankataria
Jun 4, 2023

Hi, I'm having issues running KafkaConnect with Strimzi.
It ran fine initially, but recently started seeing more restarts at POD level. I tried to increase the replicas of KafkaConnect so the connectors won't get affected but then even more frequent CrashLoopBackOffs.

Please refer to below timeline -

ISSUE 1: Frequent Restarts at POD Level of KafkaConnect
FIX: Increased Replicas so connectors won't get affected

ISSUE2: Because of Increased Replicas started seeing CrashLoopBackOffs / OOMKilled.. etc.
FIX: Googled around.. Figured its due to resource utilization, Increased the Resources and stopped seeing OOMKilled but..

ISSUE3:

The Kafka Connect cluster is still going through multiple RESTARTS,
Connectors are in Not ready state (another pain point since there is no direct way to restart connectors manually Im aware of)

FIX: Restarted the Strimzi Operator, and then Kafka Cluster

ISSUE4: Now Kafka Clusters are in Pending State from last 12hours
FIX: Post the timeline + issue in strimzi repo

Kafka Connect Config with Strimzi -

apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaConnect
metadata:
  name: pars-kc
  annotations:
    strimzi.io/use-connector-resources: "true"

spec:
  version: 3.4.0
  replicas: 3
  resources:
    limits:
      cpu: "8"
      memory: "32Gi"
    requests:
      cpu: "4"
      memory: "16Gi"
  bootstrapServers: evh-pars-dev-centralus.servicebus.windows.net:9093
  authentication:
    type: plain
    username: $ConnectionString
    passwordSecret:
      secretName: eventhubssecret
      password: eventhubspassword
  tls:
    trustedCertificates: []

  config:
    group.Id: 123
    offset.storage.topic: strimzi-connect-cluster-offsets
    config.storage.topic: strimzi-connect-cluster-configs
    status.storage.topic: strimzi-connect-cluster-status
    key.converter: org.apache.kafka.connect.json.JsonConverter
    value.converter: org.apache.kafka.connect.json.JsonConverter
    key.converter.schemas.enable: false
    value.converter.schemas.enable: true
    offset.flush.internal.ms: 10000
    request.timeout.ms: 60000
    kafka.heap.opts: -Xms1G -Xmx2G

  template:
    spec:
      containers:
        - name: pars-kc
          ports:
          - containerPort: 8083
          env:
          - name: HTTP_PROXY
            value: null
          - name: HTTPS_PROXY
            value: null

  build:
    output:
      type: docker
      image: acrparsdevcentralus.azurecr.io/connect:2.2
      pushSecret: acr-secret-2
    plugins:
      - name: snowflake-kafka-connector
        artifacts:
          - type: maven
            repository: https://repo1.maven.org/maven2/
            group: com.snowflake
            artifact: snowflake-kafka-connector
            version: 1.9.2
      - name: bc-fips
        artifacts:
          - type: maven
            repository: https://repo1.maven.org/maven2/
            group: org.bouncycastle
            artifact: bc-fips
            version: 1.0.1
      - name: bcpkix-fips
        artifacts:
          - type: maven
            repository: https://repo1.maven.org/maven2/
            group: org.bouncycastle
            artifact: bcpkix-fips
            version: 1.0.3
      - name: kafka-connect-jdbc
        artifacts:
          - type: jar
            url: https://packages.confluent.io/maven/io/confluent/kafka-connect-jdbc/10.7.0/kafka-connect-jdbc-10.7.0.jar
            version: 10.7.0
          - type: maven
            repository: https://repo1.maven.org/maven2/
            group: com.microsoft.sqlserver
            artifact: mssql-jdbc
            version: 12.2.0.jre11

  template:
    pod:
      imagePullSecrets:
        - name: acr-secret-2

Kubernetes Events -


LAST SEEN   TYPE      REASON              OBJECT                                 MESSAGE
3m44s       Normal    NotTriggerScaleUp   pod/ccat-connect-5b5cfbfcf6-q7crh      pod didn't trigger scale-up: 1 Insufficient cpu, 1 Insufficient memory
23m         Normal    NotTriggerScaleUp   pod/ccat-connect-5b5cfbfcf6-q7crh      pod didn't trigger scale-up: 1 Insufficient memory, 1 Insufficient cpu
4m31s       Warning   FailedScheduling    pod/ccat-connect-5b5cfbfcf6-q7crh      0/1 nodes are available: 1 Insufficient cpu, 1 Insufficient memory. preemption: 0/1 nodes are available: 1 No preemption victims found for incoming pod.
23m         Normal    NotTriggerScaleUp   pod/ccat-connect-85f5c7455d-d4j9c      pod didn't trigger scale-up: 1 Insufficient memory, 1 Insufficient cpu
3m44s       Normal    NotTriggerScaleUp   pod/ccat-connect-85f5c7455d-d4j9c      pod didn't trigger scale-up: 1 Insufficient cpu, 1 Insufficient memory
4m31s       Warning   FailedScheduling    pod/ccat-connect-85f5c7455d-d4j9c      0/1 nodes are available: 1 Insufficient cpu, 1 Insufficient memory. preemption: 0/1 nodes are available: 1 No preemption victims found for incoming pod.
3m44s       Normal    NotTriggerScaleUp   pod/ccat-connect-85f5c7455d-jp7sd      pod didn't trigger scale-up: 1 Insufficient cpu, 1 Insufficient memory
13m         Normal    NotTriggerScaleUp   pod/ccat-connect-85f5c7455d-jp7sd      pod didn't trigger scale-up: 1 Insufficient memory, 1 Insufficient cpu
4m31s       Warning   FailedScheduling    pod/ccat-connect-85f5c7455d-jp7sd      0/1 nodes are available: 1 Insufficient cpu, 1 Insufficient memory. preemption: 0/1 nodes are available: 1 No preemption victims found for incoming pod.
3m44s       Normal    NotTriggerScaleUp   pod/ccat-connect-85f5c7455d-mz9tf      pod didn't trigger scale-up: 1 Insufficient cpu, 1 Insufficient memory
4m31s       Warning   FailedScheduling    pod/ccat-connect-85f5c7455d-mz9tf      0/1 nodes are available: 1 Insufficient cpu, 1 Insufficient memory. preemption: 0/1 nodes are available: 1 No preemption victims found for incoming pod.
3m54s       Normal    NotTriggerScaleUp   pod/pars-kc-connect-787c55f74f-6b4xk   pod didn't trigger scale-up: 1 Insufficient cpu, 1 Insufficient memory
4m31s       Warning   FailedScheduling    pod/pars-kc-connect-787c55f74f-6b4xk   0/1 nodes are available: 1 Insufficient cpu, 1 Insufficient memory. preemption: 0/1 nodes are available: 1 No preemption victims found for incoming pod.
3m54s       Normal    NotTriggerScaleUp   pod/pars-kc-connect-787c55f74f-g4twt   pod didn't trigger scale-up: 1 Insufficient cpu, 1 Insufficient memory
18m         Normal    NotTriggerScaleUp   pod/pars-kc-connect-787c55f74f-g4twt   pod didn't trigger scale-up: 1 Insufficient memory, 1 Insufficient cpu
4m31s       Warning   FailedScheduling    pod/pars-kc-connect-787c55f74f-g4twt   0/1 nodes are available: 1 Insufficient cpu, 1 Insufficient memory. preemption: 0/1 nodes are available: 1 No preemption victims found for incoming pod.
3m54s       Normal    NotTriggerScaleUp   pod/pars-kc-connect-787c55f74f-lzhqx   pod didn't trigger scale-up: 1 Insufficient cpu, 1 Insufficient memory
4m31s       Warning   FailedScheduling    pod/pars-kc-connect-787c55f74f-lzhqx   0/1 nodes are available: 1 Insufficient cpu, 1 Insufficient memory. preemption: 0/1 nodes are available: 1 No preemption victims found for incoming pod.

Error Logs -
Logs Posted at - https://www.logpasta.com/paste/a8279626-1eb6-4fe2-a701-20a6f26d0c13
I see a lot of reconciliation errors, why is that? I read at some places its due to proxy issues, I tried to fix that with HTTP / HTTPS with null values but in the end setup is still not working

2023-06-04 21:13:49 ERROR Util:166 - Reconciliation #830(timer) KafkaConnect(kafka-connect/pars-kc): Exceeded timeout of 300000ms while waiting for Deployment resource pars-kc-connect in namespace kafka-connect to be ready
2023-06-04 21:13:49 ERROR AbstractOperator:260 - Reconciliation #830(timer) KafkaConnect(kafka-connect/pars-kc): createOrUpdate failed
2023-06-04 21:13:49 ERROR Util:166 - Reconciliation #831(timer) KafkaConnect(kafka-connect/ccat): Exceeded timeout of 300000ms while waiting for Deployment resource ccat-connect in namespace kafka-connect to be ready
2023-06-04 21:13:49 ERROR AbstractOperator:260 - Reconciliation #831(timer) KafkaConnect(kafka-connect/ccat): createOrUpdate failed
2023-06-04 21:19:49 ERROR Util:166 - Reconciliation #836(timer) KafkaConnect(kafka-connect/pars-kc): Exceeded timeout of 300000ms while waiting for Deployment resource pars-kc-connect in namespace kafka-connect to be ready
2023-06-04 21:19:49 ERROR AbstractOperator:260 - Reconciliation #836(timer) KafkaConnect(kafka-connect/pars-kc): createOrUpdate failed
2023-06-04 21:19:49 ERROR Util:166 - Reconciliation #837(timer) KafkaConnect(kafka-connect/ccat): Exceeded timeout of 300000ms while waiting for Deployment resource ccat-connect in namespace kafka-connect to be ready
2023-06-04 21:19:49 ERROR AbstractOperator:260 - Reconciliation #837(timer) KafkaConnect(kafka-connect/ccat): createOrUpdate failed
2023-06-04 21:24:02 ERROR InformerUtils:29 - Caught exception in the Pod informer which is started
	at io.fabric8.kubernetes.client.dsl.internal.WatcherWebSocketListener.onError(WatcherWebSocketListener.java:51) ~[io.fabric8.kubernetes-client-6.6.1.jar:?]
	at io.fabric8.kubernetes.client.jdkhttp.JdkWebSocketImpl$ListenerAdapter.onError(JdkWebSocketImpl.java:85) ~[io.fabric8.kubernetes-httpclient-jdk-6.6.1.jar:?]
	at jdk.internal.net.http.websocket.WebSocketImpl$ReceiveTask.processError(WebSocketImpl.java:508) ~[java.net.http:?]
	at io.fabric8.kubernetes.client.dsl.internal.WatcherWebSocketListener.onError(WatcherWebSocketListener.java:51) ~[io.fabric8.kubernetes-client-6.6.1.jar:?]
	at io.fabric8.kubernetes.client.jdkhttp.JdkWebSocketImpl$ListenerAdapter.onError(JdkWebSocketImpl.java:85) ~[io.fabric8.kubernetes-httpclient-jdk-6.6.1.jar:?]
	at jdk.internal.net.http.websocket.WebSocketImpl$ReceiveTask.processError(WebSocketImpl.java:508) ~[java.net.http:?]
2023-06-04 21:25:49 ERROR Util:166 - Reconciliation #842(timer) KafkaConnect(kafka-connect/pars-kc): Exceeded timeout of 300000ms while waiting for Deployment resource pars-kc-connect in namespace kafka-connect to be ready
2023-06-04 21:25:49 ERROR AbstractOperator:260 - Reconciliation #842(timer) KafkaConnect(kafka-connect/pars-kc): createOrUpdate failed
2023-06-04 21:25:49 ERROR Util:166 - Reconciliation #843(timer) KafkaConnect(kafka-connect/ccat): Exceeded timeout of 300000ms while waiting for Deployment resource ccat-connect in namespace kafka-connect to be ready
2023-06-04 21:25:49 ERROR AbstractOperator:260 - Reconciliation #843(timer) KafkaConnect(kafka-connect/ccat): createOrUpdate failed
2023-06-04 21:31:49 ERROR Util:166 - Reconciliation #848(timer) KafkaConnect(kafka-connect/pars-kc): Exceeded timeout of 300000ms while waiting for Deployment resource pars-kc-connect in namespace kafka-connect to be ready
2023-06-04 21:31:49 ERROR AbstractOperator:260 - Reconciliation #848(timer) KafkaConnect(kafka-connect/pars-kc): createOrUpdate failed
2023-06-04 21:31:49 ERROR Util:166 - Reconciliation #849(timer) KafkaConnect(kafka-connect/ccat): Exceeded timeout of 300000ms while waiting for Deployment resource ccat-connect in namespace kafka-connect to be ready
2023-06-04 21:31:49 ERROR AbstractOperator:260 - Reconciliation #849(timer) KafkaConnect(kafka-connect/ccat): createOrUpdate failed
2023-06-04 21:31:52 ERROR InformerUtils:29 - Caught exception in the KafkaMirrorMaker2 informer which is started
	at io.fabric8.kubernetes.client.dsl.internal.WatcherWebSocketListener.onError(WatcherWebSocketListener.java:51) ~[io.fabric8.kubernetes-client-6.6.1.jar:?]
	at io.fabric8.kubernetes.client.jdkhttp.JdkWebSocketImpl$ListenerAdapter.onError(JdkWebSocketImpl.java:85) ~[io.fabric8.kubernetes-httpclient-jdk-6.6.1.jar:?]
	at jdk.internal.net.http.websocket.WebSocketImpl$ReceiveTask.processError(WebSocketImpl.java:508) ~[java.net.http:?]
	at io.fabric8.kubernetes.client.dsl.internal.WatcherWebSocketListener.onError(WatcherWebSocketListener.java:51) ~[io.fabric8.kubernetes-client-6.6.1.jar:?]
	at io.fabric8.kubernetes.client.jdkhttp.JdkWebSocketImpl$ListenerAdapter.onError(JdkWebSocketImpl.java:85) ~[io.fabric8.kubernetes-httpclient-jdk-6.6.1.jar:?]
	at jdk.internal.net.http.websocket.WebSocketImpl$ReceiveTask.processError(WebSocketImpl.java:508) ~[java.net.http:?]
2023-06-04 21:37:49 ERROR Util:166 - Reconciliation #854(timer) KafkaConnect(kafka-connect/pars-kc): Exceeded timeout of 300000ms while waiting for Deployment resource pars-kc-connect in namespace kafka-connect to be ready
2023-06-04 21:37:49 ERROR AbstractOperator:260 - Reconciliation #854(timer) KafkaConnect(kafka-connect/pars-kc): createOrUpdate failed
2023-06-04 21:37:49 ERROR Util:166 - Reconciliation #855(timer) KafkaConnect(kafka-connect/ccat): Exceeded timeout of 300000ms while waiting for Deployment resource ccat-connect in namespace kafka-connect to be ready
2023-06-04 21:37:49 ERROR AbstractOperator:260 - Reconciliation #855(timer) KafkaConnect(kafka-connect/ccat): createOrUpdate failed
2023-06-04 21:43:49 ERROR Util:166 - Reconciliation #860(timer) KafkaConnect(kafka-connect/pars-kc): Exceeded timeout of 300000ms while waiting for Deployment resource pars-kc-connect in namespace kafka-connect to be ready
2023-06-04 21:43:49 ERROR AbstractOperator:260 - Reconciliation #860(timer) KafkaConnect(kafka-connect/pars-kc): createOrUpdate failed
2023-06-04 21:43:49 ERROR Util:166 - Reconciliation #861(timer) KafkaConnect(kafka-connect/ccat): Exceeded timeout of 300000ms while waiting for Deployment resource ccat-connect in namespace kafka-connect to be ready
2023-06-04 21:43:49 ERROR AbstractOperator:260 - Reconciliation #861(timer) KafkaConnect(kafka-connect/ccat): createOrUpdate failed
2023-06-04 21:46:25 ERROR InformerUtils:29 - Caught exception in the KafkaConnect informer which is started
	at io.fabric8.kubernetes.client.dsl.internal.WatcherWebSocketListener.onError(WatcherWebSocketListener.java:51) ~[io.fabric8.kubernetes-client-6.6.1.jar:?]
	at io.fabric8.kubernetes.client.jdkhttp.JdkWebSocketImpl$ListenerAdapter.onError(JdkWebSocketImpl.java:85) ~[io.fabric8.kubernetes-httpclient-jdk-6.6.1.jar:?]
	at jdk.internal.net.http.websocket.WebSocketImpl$ReceiveTask.processError(WebSocketImpl.java:508) ~[java.net.http:?]
	at io.fabric8.kubernetes.client.dsl.internal.WatcherWebSocketListener.onError(WatcherWebSocketListener.java:51) ~[io.fabric8.kubernetes-client-6.6.1.jar:?]
	at io.fabric8.kubernetes.client.jdkhttp.JdkWebSocketImpl$ListenerAdapter.onError(JdkWebSocketImpl.java:85) ~[io.fabric8.kubernetes-httpclient-jdk-6.6.1.jar:?]
	at jdk.internal.net.http.websocket.WebSocketImpl$ReceiveTask.processError(WebSocketImpl.java:508) ~[java.net.http:?]
2023-06-04 21:49:49 ERROR Util:166 - Reconciliation #866(timer) KafkaConnect(kafka-connect/pars-kc): Exceeded timeout of 300000ms while waiting for Deployment resource pars-kc-connect in namespace kafka-connect to be ready
2023-06-04 21:49:49 ERROR AbstractOperator:260 - Reconciliation #866(timer) KafkaConnect(kafka-connect/pars-kc): createOrUpdate failed
2023-06-04 21:49:49 ERROR Util:166 - Reconciliation #867(timer) KafkaConnect(kafka-connect/ccat): Exceeded timeout of 300000ms while waiting for Deployment resource ccat-connect in namespace kafka-connect to be ready
2023-06-04 21:49:49 ERROR AbstractOperator:260 - Reconciliation #867(timer) KafkaConnect(kafka-connect/ccat): createOrUpdate failed
2023-06-04 21:50:50 ERROR InformerUtils:29 - Caught exception in the KafkaConnect informer which is started
	at io.fabric8.kubernetes.client.dsl.internal.WatcherWebSocketListener.onError(WatcherWebSocketListener.java:51) ~[io.fabric8.kubernetes-client-6.6.1.jar:?]
	at io.fabric8.kubernetes.client.jdkhttp.JdkWebSocketImpl$ListenerAdapter.onError(JdkWebSocketImpl.java:85) ~[io.fabric8.kubernetes-httpclient-jdk-6.6.1.jar:?]
	at jdk.internal.net.http.websocket.WebSocketImpl$ReceiveTask.processError(WebSocketImpl.java:508) ~[java.net.http:?]
	at io.fabric8.kubernetes.client.dsl.internal.WatcherWebSocketListener.onError(WatcherWebSocketListener.java:51) ~[io.fabric8.kubernetes-client-6.6.1.jar:?]
	at io.fabric8.kubernetes.client.jdkhttp.JdkWebSocketImpl$ListenerAdapter.onError(JdkWebSocketImpl.java:85) ~[io.fabric8.kubernetes-httpclient-jdk-6.6.1.jar:?]
	at jdk.internal.net.http.websocket.WebSocketImpl$ReceiveTask.processError(WebSocketImpl.java:508) ~[java.net.http:?]
2023-06-04 21:52:29 ERROR InformerUtils:29 - Caught exception in the KafkaMirrorMaker2 informer which is started
	at io.fabric8.kubernetes.client.dsl.internal.WatcherWebSocketListener.onError(WatcherWebSocketListener.java:51) ~[io.fabric8.kubernetes-client-6.6.1.jar:?]
	at io.fabric8.kubernetes.client.jdkhttp.JdkWebSocketImpl$ListenerAdapter.onError(JdkWebSocketImpl.java:85) ~[io.fabric8.kubernetes-httpclient-jdk-6.6.1.jar:?]
	at jdk.internal.net.http.websocket.WebSocketImpl$ReceiveTask.processError(WebSocketImpl.java:508) ~[java.net.http:?]
	at io.fabric8.kubernetes.client.dsl.internal.WatcherWebSocketListener.onError(WatcherWebSocketListener.java:51) ~[io.fabric8.kubernetes-client-6.6.1.jar:?]
	at io.fabric8.kubernetes.client.jdkhttp.JdkWebSocketImpl$ListenerAdapter.onError(JdkWebSocketImpl.java:85) ~[io.fabric8.kubernetes-httpclient-jdk-6.6.1.jar:?]
	at jdk.internal.net.http.websocket.WebSocketImpl$ReceiveTask.processError(WebSocketImpl.java:508) ~[java.net.http:?]
2023-06-04 21:55:19 ERROR InformerUtils:29 - Caught exception in the KafkaConnect informer which is started
	at io.fabric8.kubernetes.client.dsl.internal.WatcherWebSocketListener.onError(WatcherWebSocketListener.java:51) ~[io.fabric8.kubernetes-client-6.6.1.jar:?]
	at io.fabric8.kubernetes.client.jdkhttp.JdkWebSocketImpl$ListenerAdapter.onError(JdkWebSocketImpl.java:85) ~[io.fabric8.kubernetes-httpclient-jdk-6.6.1.jar:?]
	at jdk.internal.net.http.websocket.WebSocketImpl$ReceiveTask.processError(WebSocketImpl.java:508) ~[java.net.http:?]
	at io.fabric8.kubernetes.client.dsl.internal.WatcherWebSocketListener.onError(WatcherWebSocketListener.java:51) ~[io.fabric8.kubernetes-client-6.6.1.jar:?]
	at io.fabric8.kubernetes.client.jdkhttp.JdkWebSocketImpl$ListenerAdapter.onError(JdkWebSocketImpl.java:85) ~[io.fabric8.kubernetes-httpclient-jdk-6.6.1.jar:?]
	at jdk.internal.net.http.websocket.WebSocketImpl$ReceiveTask.processError(WebSocketImpl.java:508) ~[java.net.http:?]
2023-06-04 21:55:49 ERROR Util:166 - Reconciliation #872(timer) KafkaConnect(kafka-connect/pars-kc): Exceeded timeout of 300000ms while waiting for Deployment resource pars-kc-connect in namespace kafka-connect to be ready
2023-06-04 21:55:49 ERROR AbstractOperator:260 - Reconciliation #872(timer) KafkaConnect(kafka-connect/pars-kc): createOrUpdate failed
2023-06-04 21:55:49 ERROR Util:166 - Reconciliation #873(timer) KafkaConnect(kafka-connect/ccat): Exceeded timeout of 300000ms while waiting for Deployment resource ccat-connect in namespace kafka-connect to be ready
2023-06-04 21:55:49 ERROR AbstractOperator:260 - Reconciliation #873(timer) KafkaConnect(kafka-connect/ccat): createOrUpdate failed
2023-06-04 21:57:07 ERROR AbstractOperator:492 - Watcher closed with exception in namespace kafka-connect
	at io.fabric8.kubernetes.client.dsl.internal.WatcherWebSocketListener.onError(WatcherWebSocketListener.java:51) ~[io.fabric8.kubernetes-client-6.6.1.jar:?]
	at io.fabric8.kubernetes.client.jdkhttp.JdkWebSocketImpl$ListenerAdapter.onError(JdkWebSocketImpl.java:85) ~[io.fabric8.kubernetes-httpclient-jdk-6.6.1.jar:?]
	at jdk.internal.net.http.websocket.WebSocketImpl$ReceiveTask.processError(WebSocketImpl.java:508) ~[java.net.http:?]
2023-06-04 22:01:49 ERROR Util:166 - Reconciliation #878(timer) KafkaConnect(kafka-connect/pars-kc): Exceeded timeout of 300000ms while waiting for Deployment resource pars-kc-connect in namespace kafka-connect to be ready
2023-06-04 22:01:49 ERROR AbstractOperator:260 - Reconciliation #878(timer) KafkaConnect(kafka-connect/pars-kc): createOrUpdate failed
2023-06-04 22:01:49 ERROR Util:166 - Reconciliation #879(timer) KafkaConnect(kafka-connect/ccat): Exceeded timeout of 300000ms while waiting for Deployment resource ccat-connect in namespace kafka-connect to be ready
2023-06-04 22:01:49 ERROR AbstractOperator:260 - Reconciliation #879(timer) KafkaConnect(kafka-connect/ccat): createOrUpdate failed
2023-06-04 22:03:53 ERROR InformerUtils:29 - Caught exception in the KafkaMirrorMaker2 informer which is started
	at io.fabric8.kubernetes.client.dsl.internal.WatcherWebSocketListener.onError(WatcherWebSocketListener.java:51) ~[io.fabric8.kubernetes-client-6.6.1.jar:?]
	at io.fabric8.kubernetes.client.jdkhttp.JdkWebSocketImpl$ListenerAdapter.onError(JdkWebSocketImpl.java:85) ~[io.fabric8.kubernetes-httpclient-jdk-6.6.1.jar:?]
	at jdk.internal.net.http.websocket.WebSocketImpl$ReceiveTask.processError(WebSocketImpl.java:508) ~[java.net.http:?]
	at io.fabric8.kubernetes.client.dsl.internal.WatcherWebSocketListener.onError(WatcherWebSocketListener.java:51) ~[io.fabric8.kubernetes-client-6.6.1.jar:?]
	at io.fabric8.kubernetes.client.jdkhttp.JdkWebSocketImpl$ListenerAdapter.onError(JdkWebSocketImpl.java:85) ~[io.fabric8.kubernetes-httpclient-jdk-6.6.1.jar:?]
	at jdk.internal.net.http.websocket.WebSocketImpl$ReceiveTask.processError(WebSocketImpl.java:508) ~[java.net.http:?]

Answered by scholzj

Jun 5, 2023

Your Pod is pending because you don't have resources for it: 0/1 nodes are available: 1 Insufficient cpu, 1 Insufficient memory. preemption: 0/1 nodes are available: 1 No preemption victims found for incoming pod.. You need to make sure your cluster has enough resources to accommodate the Pod which does not seem to be the case.

Setting a different memory request and limit is also quite a bad practice for something like Java. You should ideally use the same value for both.

You did not share any logs from the Kafka Connect pod when it is running. So it is not clear what is going on there.

View full answer

scholzj · 2023-06-04T23:04:16Z

scholzj
Jun 4, 2023
Maintainer

Please format the YAML as a code to make it readable. You will also need to share the full logs (at least from the Strimzi Cluster operator & the Connect pods) - without them, nobody can tell what is going on in your cluster. If you have pods in the pending state, then you should probably also share the Kubernetes events which will explain why it is pending - but in general, that is most often caused by infrastructure/environment issues.

You should probably also pick up one state and focus on it not 4 different ones.

14 replies

imrohankataria Jun 5, 2023
Author

Reduced the replicas to 1 from 3, deleted cluster deployments and re-installed them via apply -f
Build process of image was a success, and pod runs successfully but when it comes to running the actual kafkaconnect pod it becomed cold turkey and stays in Pending state

NAME                                        READY   STATUS    RESTARTS   AGE
ccat-connect-58755fcc8b-d4rbk               0/1     Pending   0          82s
debezium-ui-8685fcd5f7-ztxzh                1/1     Running   0          18h
pars-kc-connect-build                       1/1     Running   0          17s
strimzi-cluster-operator-5cddf5b479-gvfgj   1/1     Running   0          17h

Events -

LAST SEEN   TYPE      REASON              OBJECT                                  MESSAGE
16m         Warning   FailedScheduling    pod/ccat-connect-547f5b8d97-qq89r       0/1 nodes are available: 1 Insufficient cpu, 1 Insufficient memory. preemption: 0/1 nodes are available: 1 No preemption victims found for incoming pod.
11m         Normal    NotTriggerScaleUp   pod/ccat-connect-547f5b8d97-qq89r       pod didn't trigger scale-up: 1 Insufficient cpu, 1 Insufficient memory
11m         Warning   FailedScheduling    pod/ccat-connect-547f5b8d97-qq89r       0/1 nodes are available: 1 Insufficient cpu, 1 Insufficient memory. preemption: 0/1 nodes are available: 1 No preemption victims found for incoming pod.
9m23s       Warning   FailedScheduling    pod/ccat-connect-547f5b8d97-qq89r       skip schedule deleting pod: kafka-connect/ccat-connect-547f5b8d97-qq89r
16m         Normal    SuccessfulCreate    replicaset/ccat-connect-547f5b8d97      Created pod: ccat-connect-547f5b8d97-qq89r
4m49s       Warning   FailedScheduling    pod/ccat-connect-58755fcc8b-d4rbk       0/1 nodes are available: 1 Insufficient cpu, 1 Insufficient memory. preemption: 0/1 nodes are available: 1 No preemption victims found for incoming pod.
4m45s       Normal    NotTriggerScaleUp   pod/ccat-connect-58755fcc8b-d4rbk       pod didn't trigger scale-up: 1 Insufficient cpu, 1 Insufficient memory
40s         Warning   FailedScheduling    pod/ccat-connect-58755fcc8b-d4rbk       0/1 nodes are available: 1 Insufficient cpu, 1 Insufficient memory. preemption: 0/1 nodes are available: 1 No preemption victims found for incoming pod.
4m49s       Normal    SuccessfulCreate    replicaset/ccat-connect-58755fcc8b      Created pod: ccat-connect-58755fcc8b-d4rbk
22m         Normal    NotTriggerScaleUp   pod/ccat-connect-5b5cfbfcf6-q7crh       pod didn't trigger scale-up: 1 Insufficient cpu, 1 Insufficient memory
17m         Normal    NotTriggerScaleUp   pod/ccat-connect-5b5cfbfcf6-q7crh       pod didn't trigger scale-up: 1 Insufficient memory, 1 Insufficient cpu
19m         Warning   FailedScheduling    pod/ccat-connect-5b5cfbfcf6-q7crh       0/1 nodes are available: 1 Insufficient cpu, 1 Insufficient memory. preemption: 0/1 nodes are available: 1 No preemption victims found for incoming pod.
16m         Warning   FailedScheduling    pod/ccat-connect-5b5cfbfcf6-q7crh       skip schedule deleting pod: kafka-connect/ccat-connect-5b5cfbfcf6-q7crh
16m         Normal    SuccessfulDelete    replicaset/ccat-connect-5b5cfbfcf6      Deleted pod: ccat-connect-5b5cfbfcf6-q7crh
25m         Warning   FailedScheduling    pod/ccat-connect-76df65449f-2n2n7       0/1 nodes are available: 1 Insufficient cpu, 1 Insufficient memory. preemption: 0/1 nodes are available: 1 No preemption victims found for incoming pod.
10m         Normal    NotTriggerScaleUp   pod/ccat-connect-76df65449f-2n2n7       pod didn't trigger scale-up: 1 Insufficient cpu, 1 Insufficient memory
20m         Warning   FailedScheduling    pod/ccat-connect-76df65449f-2n2n7       0/1 nodes are available: 1 Insufficient cpu, 1 Insufficient memory. preemption: 0/1 nodes are available: 1 No preemption victims found for incoming pod.
9m23s       Warning   FailedScheduling    pod/ccat-connect-76df65449f-2n2n7       skip schedule deleting pod: kafka-connect/ccat-connect-76df65449f-2n2n7
25m         Normal    SuccessfulCreate    replicaset/ccat-connect-76df65449f      Created pod: ccat-connect-76df65449f-2n2n7
48m         Normal    NotTriggerScaleUp   pod/ccat-connect-85f5c7455d-d4j9c       pod didn't trigger scale-up: 1 Insufficient cpu, 1 Insufficient memory
48m         Warning   FailedScheduling    pod/ccat-connect-85f5c7455d-d4j9c       0/1 nodes are available: 1 Insufficient cpu, 1 Insufficient memory. preemption: 0/1 nodes are available: 1 No preemption victims found for incoming pod.
45m         Warning   FailedScheduling    pod/ccat-connect-85f5c7455d-d4j9c       skip schedule deleting pod: kafka-connect/ccat-connect-85f5c7455d-d4j9c
38m         Normal    NotTriggerScaleUp   pod/ccat-connect-85f5c7455d-jp7sd       pod didn't trigger scale-up: 1 Insufficient cpu, 1 Insufficient memory
37m         Warning   FailedScheduling    pod/ccat-connect-85f5c7455d-jp7sd       0/1 nodes are available: 1 Insufficient cpu, 1 Insufficient memory. preemption: 0/1 nodes are available: 1 No preemption victims found for incoming pod.
35m         Warning   FailedScheduling    pod/ccat-connect-85f5c7455d-jp7sd       skip schedule deleting pod: kafka-connect/ccat-connect-85f5c7455d-jp7sd
28m         Normal    NotTriggerScaleUp   pod/ccat-connect-85f5c7455d-mz9tf       pod didn't trigger scale-up: 1 Insufficient cpu, 1 Insufficient memory
30m         Warning   FailedScheduling    pod/ccat-connect-85f5c7455d-mz9tf       0/1 nodes are available: 1 Insufficient cpu, 1 Insufficient memory. preemption: 0/1 nodes are available: 1 No preemption victims found for incoming pod.
25m         Warning   FailedScheduling    pod/ccat-connect-85f5c7455d-mz9tf       skip schedule deleting pod: kafka-connect/ccat-connect-85f5c7455d-mz9tf
45m         Normal    SuccessfulDelete    replicaset/ccat-connect-85f5c7455d      Deleted pod: ccat-connect-85f5c7455d-d4j9c
35m         Normal    SuccessfulDelete    replicaset/ccat-connect-85f5c7455d      Deleted pod: ccat-connect-85f5c7455d-jp7sd
25m         Normal    SuccessfulDelete    replicaset/ccat-connect-85f5c7455d      Deleted pod: ccat-connect-85f5c7455d-mz9tf
7m44s       Normal    Scheduled           pod/ccat-connect-build                  Successfully assigned kafka-connect/ccat-connect-build to aks-agentpool-38288350-vmss000000
7m44s       Normal    Pulled              pod/ccat-connect-build                  Container image "quay.io/strimzi/kaniko-executor:0.35.0" already present on machine
7m44s       Normal    Created             pod/ccat-connect-build                  Created container ccat-connect-build
7m44s       Normal    Started             pod/ccat-connect-build                  Started container ccat-connect-build
45m         Normal    ScalingReplicaSet   deployment/ccat-connect                 Scaled down replica set ccat-connect-85f5c7455d to 2
35m         Normal    ScalingReplicaSet   deployment/ccat-connect                 Scaled down replica set ccat-connect-85f5c7455d to 1
25m         Normal    ScalingReplicaSet   deployment/ccat-connect                 Scaled down replica set ccat-connect-85f5c7455d to 0
25m         Normal    ScalingReplicaSet   deployment/ccat-connect                 Scaled up replica set ccat-connect-76df65449f to 1
16m         Normal    ScalingReplicaSet   deployment/ccat-connect                 Scaled down replica set ccat-connect-5b5cfbfcf6 to 0
16m         Normal    ScalingReplicaSet   deployment/ccat-connect                 Scaled up replica set ccat-connect-547f5b8d97 to 1
4m49s       Normal    NoPods              poddisruptionbudget/ccat-connect        No matching pods found
4m49s       Normal    ScalingReplicaSet   deployment/ccat-connect                 Scaled up replica set ccat-connect-58755fcc8b to 1
48m         Normal    Scheduled           pod/debezium-ui-7dc4b5cb84-k7bq6        Successfully assigned kafka-connect/debezium-ui-7dc4b5cb84-k7bq6 to aks-agentpool-38288350-vmss000000
48m         Normal    Pulling             pod/debezium-ui-7dc4b5cb84-k7bq6        Pulling image "debezium/debezium-ui:latest"
48m         Normal    Pulled              pod/debezium-ui-7dc4b5cb84-k7bq6        Successfully pulled image "debezium/debezium-ui:latest" in 451.876677ms
48m         Normal    Created             pod/debezium-ui-7dc4b5cb84-k7bq6        Created container debezium-ui
48m         Normal    Started             pod/debezium-ui-7dc4b5cb84-k7bq6        Started container debezium-ui
48m         Normal    Killing             pod/debezium-ui-7dc4b5cb84-k7bq6        Stopping container debezium-ui
8m40s       Normal    Scheduled           pod/debezium-ui-7dc4b5cb84-m24b8        Successfully assigned kafka-connect/debezium-ui-7dc4b5cb84-m24b8 to aks-agentpool-38288350-vmss000000
8m40s       Normal    Pulling             pod/debezium-ui-7dc4b5cb84-m24b8        Pulling image "debezium/debezium-ui:latest"
8m39s       Normal    Pulled              pod/debezium-ui-7dc4b5cb84-m24b8        Successfully pulled image "debezium/debezium-ui:latest" in 477.750443ms
8m39s       Normal    Created             pod/debezium-ui-7dc4b5cb84-m24b8        Created container debezium-ui
8m39s       Normal    Started             pod/debezium-ui-7dc4b5cb84-m24b8        Started container debezium-ui
8m38s       Normal    Killing             pod/debezium-ui-7dc4b5cb84-m24b8        Stopping container debezium-ui
30m         Normal    Scheduled           pod/debezium-ui-7dc4b5cb84-qznw8        Successfully assigned kafka-connect/debezium-ui-7dc4b5cb84-qznw8 to aks-agentpool-38288350-vmss000000
30m         Normal    Pulling             pod/debezium-ui-7dc4b5cb84-qznw8        Pulling image "debezium/debezium-ui:latest"
30m         Normal    Pulled              pod/debezium-ui-7dc4b5cb84-qznw8        Successfully pulled image "debezium/debezium-ui:latest" in 452.617433ms
30m         Normal    Created             pod/debezium-ui-7dc4b5cb84-qznw8        Created container debezium-ui
30m         Normal    Started             pod/debezium-ui-7dc4b5cb84-qznw8        Started container debezium-ui
30m         Normal    Killing             pod/debezium-ui-7dc4b5cb84-qznw8        Stopping container debezium-ui
48m         Normal    SuccessfulCreate    replicaset/debezium-ui-7dc4b5cb84       Created pod: debezium-ui-7dc4b5cb84-k7bq6
48m         Normal    SuccessfulDelete    replicaset/debezium-ui-7dc4b5cb84       Deleted pod: debezium-ui-7dc4b5cb84-k7bq6
30m         Normal    SuccessfulCreate    replicaset/debezium-ui-7dc4b5cb84       Created pod: debezium-ui-7dc4b5cb84-qznw8
30m         Normal    SuccessfulDelete    replicaset/debezium-ui-7dc4b5cb84       Deleted pod: debezium-ui-7dc4b5cb84-qznw8
8m40s       Normal    SuccessfulCreate    replicaset/debezium-ui-7dc4b5cb84       Created pod: debezium-ui-7dc4b5cb84-m24b8
8m40s       Normal    SuccessfulDelete    replicaset/debezium-ui-7dc4b5cb84       Deleted pod: debezium-ui-7dc4b5cb84-m24b8
8m40s       Normal    ScalingReplicaSet   deployment/debezium-ui                  Scaled up replica set debezium-ui-7dc4b5cb84 to 1
8m40s       Normal    ScalingReplicaSet   deployment/debezium-ui                  Scaled down replica set debezium-ui-7dc4b5cb84 to 0
16m         Warning   FailedScheduling    pod/pars-kc-connect-5bdfc649c7-6ksl4    0/1 nodes are available: 1 Insufficient cpu, 1 Insufficient memory. preemption: 0/1 nodes are available: 1 No preemption victims found for incoming pod.
11m         Normal    NotTriggerScaleUp   pod/pars-kc-connect-5bdfc649c7-6ksl4    pod didn't trigger scale-up: 1 Insufficient cpu, 1 Insufficient memory
11m         Warning   FailedScheduling    pod/pars-kc-connect-5bdfc649c7-6ksl4    0/1 nodes are available: 1 Insufficient cpu, 1 Insufficient memory. preemption: 0/1 nodes are available: 1 No preemption victims found for incoming pod.
9m15s       Warning   FailedScheduling    pod/pars-kc-connect-5bdfc649c7-6ksl4    skip schedule deleting pod: kafka-connect/pars-kc-connect-5bdfc649c7-6ksl4
16m         Normal    SuccessfulCreate    replicaset/pars-kc-connect-5bdfc649c7   Created pod: pars-kc-connect-5bdfc649c7-6ksl4
27m         Warning   FailedScheduling    pod/pars-kc-connect-5fd66767b-xh6b4     0/1 nodes are available: 1 Insufficient cpu, 1 Insufficient memory. preemption: 0/1 nodes are available: 1 No preemption victims found for incoming pod.
17m         Normal    NotTriggerScaleUp   pod/pars-kc-connect-5fd66767b-xh6b4     pod didn't trigger scale-up: 1 Insufficient cpu, 1 Insufficient memory
12m         Normal    NotTriggerScaleUp   pod/pars-kc-connect-5fd66767b-xh6b4     pod didn't trigger scale-up: 1 Insufficient memory, 1 Insufficient cpu
12m         Warning   FailedScheduling    pod/pars-kc-connect-5fd66767b-xh6b4     0/1 nodes are available: 1 Insufficient cpu, 1 Insufficient memory. preemption: 0/1 nodes are available: 1 No preemption victims found for incoming pod.
9m15s       Warning   FailedScheduling    pod/pars-kc-connect-5fd66767b-xh6b4     skip schedule deleting pod: kafka-connect/pars-kc-connect-5fd66767b-xh6b4
27m         Normal    SuccessfulCreate    replicaset/pars-kc-connect-5fd66767b    Created pod: pars-kc-connect-5fd66767b-xh6b4
48m         Normal    NotTriggerScaleUp   pod/pars-kc-connect-787c55f74f-6b4xk    pod didn't trigger scale-up: 1 Insufficient cpu, 1 Insufficient memory
48m         Warning   FailedScheduling    pod/pars-kc-connect-787c55f74f-6b4xk    0/1 nodes are available: 1 Insufficient cpu, 1 Insufficient memory. preemption: 0/1 nodes are available: 1 No preemption victims found for incoming pod.
45m         Warning   FailedScheduling    pod/pars-kc-connect-787c55f74f-6b4xk    skip schedule deleting pod: kafka-connect/pars-kc-connect-787c55f74f-6b4xk
48m         Normal    NotTriggerScaleUp   pod/pars-kc-connect-787c55f74f-g4twt    pod didn't trigger scale-up: 1 Insufficient cpu, 1 Insufficient memory
48m         Warning   FailedScheduling    pod/pars-kc-connect-787c55f74f-g4twt    0/1 nodes are available: 1 Insufficient cpu, 1 Insufficient memory. preemption: 0/1 nodes are available: 1 No preemption victims found for incoming pod.
45m         Warning   FailedScheduling    pod/pars-kc-connect-787c55f74f-g4twt    skip schedule deleting pod: kafka-connect/pars-kc-connect-787c55f74f-g4twt
18m         Normal    NotTriggerScaleUp   pod/pars-kc-connect-787c55f74f-lzhqx    pod didn't trigger scale-up: 1 Insufficient cpu, 1 Insufficient memory
19m         Warning   FailedScheduling    pod/pars-kc-connect-787c55f74f-lzhqx    0/1 nodes are available: 1 Insufficient cpu, 1 Insufficient memory. preemption: 0/1 nodes are available: 1 No preemption victims found for incoming pod.
16m         Warning   FailedScheduling    pod/pars-kc-connect-787c55f74f-lzhqx    skip schedule deleting pod: kafka-connect/pars-kc-connect-787c55f74f-lzhqx
45m         Normal    SuccessfulDelete    replicaset/pars-kc-connect-787c55f74f   Deleted pod: pars-kc-connect-787c55f74f-g4twt
45m         Normal    SuccessfulDelete    replicaset/pars-kc-connect-787c55f74f   Deleted pod: pars-kc-connect-787c55f74f-6b4xk
16m         Normal    SuccessfulDelete    replicaset/pars-kc-connect-787c55f74f   Deleted pod: pars-kc-connect-787c55f74f-lzhqx
40s         Warning   FailedScheduling    pod/pars-kc-connect-85c4bf9d46-mrpk6    0/1 nodes are available: 1 Insufficient cpu, 1 Insufficient memory. preemption: 0/1 nodes are available: 1 No preemption victims found for incoming pod.
34s         Normal    NotTriggerScaleUp   pod/pars-kc-connect-85c4bf9d46-mrpk6    pod didn't trigger scale-up: 1 Insufficient memory, 1 Insufficient cpu
40s         Normal    SuccessfulCreate    replicaset/pars-kc-connect-85c4bf9d46   Created pod: pars-kc-connect-85c4bf9d46-mrpk6
3m44s       Normal    Scheduled           pod/pars-kc-connect-build               Successfully assigned kafka-connect/pars-kc-connect-build to aks-agentpool-38288350-vmss000000
3m44s       Normal    Pulled              pod/pars-kc-connect-build               Container image "quay.io/strimzi/kaniko-executor:0.35.0" already present on machine
3m44s       Normal    Created             pod/pars-kc-connect-build               Created container pars-kc-connect-build
3m44s       Normal    Started             pod/pars-kc-connect-build               Started container pars-kc-connect-build
45m         Normal    ScalingReplicaSet   deployment/pars-kc-connect              Scaled down replica set pars-kc-connect-787c55f74f to 2
45m         Normal    ScalingReplicaSet   deployment/pars-kc-connect              Scaled down replica set pars-kc-connect-787c55f74f to 1
27m         Normal    ScalingReplicaSet   deployment/pars-kc-connect              Scaled up replica set pars-kc-connect-5fd66767b to 1
16m         Normal    ScalingReplicaSet   deployment/pars-kc-connect              Scaled down replica set pars-kc-connect-787c55f74f to 0
16m         Normal    ScalingReplicaSet   deployment/pars-kc-connect              Scaled up replica set pars-kc-connect-5bdfc649c7 to 1
40s         Normal    NoPods              poddisruptionbudget/pars-kc-connect     No matching pods found
40s         Normal    ScalingReplicaSet   deployment/pars-kc-connect              Scaled up replica set pars-kc-connect-85c4bf9d46 to 1

Logs -

2023-06-05 01:16:48 INFO  ClusterOperator:139 - Triggering periodic reconciliation for namespace kafka-connect
2023-06-05 01:16:48 INFO  AbstractOperator:380 - Reconciliation #1076(timer) KafkaConnect(kafka-connect/pars-kc): Reconciliation is in progress
2023-06-05 01:16:48 WARN  StatusUtils:151 - Reconciliation #1079(timer) KafkaConnect(kafka-connect/ccat): Contains object at path spec.build.plugins.artifacts with an unknown property: version
2023-06-05 01:16:48 WARN  StatusUtils:151 - Reconciliation #1079(timer) KafkaConnect(kafka-connect/ccat): Contains object at path spec.build.plugins.artifacts with an unknown property: version
2023-06-05 01:16:48 INFO  AbstractOperator:239 - Reconciliation #1079(timer) KafkaConnect(kafka-connect/ccat): KafkaConnect ccat will be checked for creation or modification
2023-06-05 01:17:03 ERROR InformerUtils:29 - Caught exception in the KafkaMirrorMaker2 informer which is started
io.fabric8.kubernetes.client.WatcherException: Could not process websocket message
        at io.fabric8.kubernetes.client.dsl.internal.WatcherWebSocketListener.onError(WatcherWebSocketListener.java:51) ~[io.fabric8.kubernetes-client-6.6.1.jar:?]
        at io.fabric8.kubernetes.client.jdkhttp.JdkWebSocketImpl$ListenerAdapter.onError(JdkWebSocketImpl.java:85) ~[io.fabric8.kubernetes-httpclient-jdk-6.6.1.jar:?]
        at jdk.internal.net.http.websocket.WebSocketImpl$ReceiveTask.processError(WebSocketImpl.java:508) ~[java.net.http:?]
        at jdk.internal.net.http.websocket.WebSocketImpl$ReceiveTask.run(WebSocketImpl.java:462) ~[java.net.http:?]
        at jdk.internal.net.http.common.SequentialScheduler$CompleteRestartableTask.run(SequentialScheduler.java:149) ~[java.net.http:?]
        at jdk.internal.net.http.common.SequentialScheduler$SchedulableTask.run(SequentialScheduler.java:230) ~[java.net.http:?]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?]
        at java.lang.Thread.run(Thread.java:833) ~[?:?]
Caused by: java.net.SocketException: Connection reset
        at sun.nio.ch.SocketChannelImpl.throwConnectionReset(SocketChannelImpl.java:394) ~[?:?]
        at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:426) ~[?:?]
        at jdk.internal.net.http.SocketTube.readAvailable(SocketTube.java:1170) ~[java.net.http:?]
        at jdk.internal.net.http.SocketTube$InternalReadPublisher$InternalReadSubscription.read(SocketTube.java:833) ~[java.net.http:?]
        at jdk.internal.net.http.SocketTube$SocketFlowTask.run(SocketTube.java:181) ~[java.net.http:?]
        at jdk.internal.net.http.common.SequentialScheduler$SchedulableTask.run(SequentialScheduler.java:230) ~[java.net.http:?]
        at jdk.internal.net.http.common.SequentialScheduler.runOrSchedule(SequentialScheduler.java:303) ~[java.net.http:?]
        at jdk.internal.net.http.common.SequentialScheduler.runOrSchedule(SequentialScheduler.java:256) ~[java.net.http:?]
        at jdk.internal.net.http.SocketTube$InternalReadPublisher$InternalReadSubscription.signalReadable(SocketTube.java:774) ~[java.net.http:?]
        at jdk.internal.net.http.SocketTube$InternalReadPublisher$ReadEvent.signalEvent(SocketTube.java:957) ~[java.net.http:?]
        at jdk.internal.net.http.SocketTube$SocketFlowEvent.handle(SocketTube.java:253) ~[java.net.http:?]
        at jdk.internal.net.http.HttpClientImpl$SelectorManager.handleEvent(HttpClientImpl.java:979) ~[java.net.http:?]
        at jdk.internal.net.http.HttpClientImpl$SelectorManager.lambda$run$3(HttpClientImpl.java:934) ~[java.net.http:?]
        at java.util.ArrayList.forEach(ArrayList.java:1511) ~[?:?]
        at jdk.internal.net.http.HttpClientImpl$SelectorManager.run(HttpClientImpl.java:934) ~[java.net.http:?]
2023-06-05 01:17:03 WARN  Reflector:150 - watch failed for kafka.strimzi.io/v1beta2/namespaces/kafka-connect/kafkamirrormaker2s, will retry
io.fabric8.kubernetes.client.WatcherException: Could not process websocket message
        at io.fabric8.kubernetes.client.dsl.internal.WatcherWebSocketListener.onError(WatcherWebSocketListener.java:51) ~[io.fabric8.kubernetes-client-6.6.1.jar:?]
        at io.fabric8.kubernetes.client.jdkhttp.JdkWebSocketImpl$ListenerAdapter.onError(JdkWebSocketImpl.java:85) ~[io.fabric8.kubernetes-httpclient-jdk-6.6.1.jar:?]
        at jdk.internal.net.http.websocket.WebSocketImpl$ReceiveTask.processError(WebSocketImpl.java:508) ~[java.net.http:?]
        at jdk.internal.net.http.websocket.WebSocketImpl$ReceiveTask.run(WebSocketImpl.java:462) ~[java.net.http:?]
        at jdk.internal.net.http.common.SequentialScheduler$CompleteRestartableTask.run(SequentialScheduler.java:149) ~[java.net.http:?]
        at jdk.internal.net.http.common.SequentialScheduler$SchedulableTask.run(SequentialScheduler.java:230) ~[java.net.http:?]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?]
        at java.lang.Thread.run(Thread.java:833) ~[?:?]
Caused by: java.net.SocketException: Connection reset
        at sun.nio.ch.SocketChannelImpl.throwConnectionReset(SocketChannelImpl.java:394) ~[?:?]
        at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:426) ~[?:?]
        at jdk.internal.net.http.SocketTube.readAvailable(SocketTube.java:1170) ~[java.net.http:?]
        at jdk.internal.net.http.SocketTube$InternalReadPublisher$InternalReadSubscription.read(SocketTube.java:833) ~[java.net.http:?]
        at jdk.internal.net.http.SocketTube$SocketFlowTask.run(SocketTube.java:181) ~[java.net.http:?]
        at jdk.internal.net.http.common.SequentialScheduler$SchedulableTask.run(SequentialScheduler.java:230) ~[java.net.http:?]
        at jdk.internal.net.http.common.SequentialScheduler.runOrSchedule(SequentialScheduler.java:303) ~[java.net.http:?]
        at jdk.internal.net.http.common.SequentialScheduler.runOrSchedule(SequentialScheduler.java:256) ~[java.net.http:?]
        at jdk.internal.net.http.SocketTube$InternalReadPublisher$InternalReadSubscription.signalReadable(SocketTube.java:774) ~[java.net.http:?]
        at jdk.internal.net.http.SocketTube$InternalReadPublisher$ReadEvent.signalEvent(SocketTube.java:957) ~[java.net.http:?]
        at jdk.internal.net.http.SocketTube$SocketFlowEvent.handle(SocketTube.java:253) ~[java.net.http:?]
        at jdk.internal.net.http.HttpClientImpl$SelectorManager.handleEvent(HttpClientImpl.java:979) ~[java.net.http:?]
        at jdk.internal.net.http.HttpClientImpl$SelectorManager.lambda$run$3(HttpClientImpl.java:934) ~[java.net.http:?]
        at java.util.ArrayList.forEach(ArrayList.java:1511) ~[?:?]
        at jdk.internal.net.http.HttpClientImpl$SelectorManager.run(HttpClientImpl.java:934) ~[java.net.http:?]
2023-06-05 01:17:48 INFO  AbstractOperator:380 - Reconciliation #1076(timer) KafkaConnect(kafka-connect/pars-kc): Reconciliation is in progress
2023-06-05 01:17:48 INFO  AbstractOperator:380 - Reconciliation #1079(timer) KafkaConnect(kafka-connect/ccat): Reconciliation is in progress
2023-06-05 01:18:48 INFO  ClusterOperator:139 - Triggering periodic reconciliation for namespace kafka-connect
2023-06-05 01:18:48 INFO  AbstractOperator:380 - Reconciliation #1076(timer) KafkaConnect(kafka-connect/pars-kc): Reconciliation is in progress
2023-06-05 01:18:48 INFO  AbstractOperator:380 - Reconciliation #1079(timer) KafkaConnect(kafka-connect/ccat): Reconciliation is in progress
2023-06-05 01:19:43 INFO  ConnectBuildOperator:244 - Reconciliation #1079(timer) KafkaConnect(kafka-connect/ccat): Build completed successfully. New image is acrparsdevcentralus.azurecr.io/connect@sha256:37375e5c2d20f43cd3df4d1c0d9814fbaff09e48a188ca64f563463933271f6a.
2023-06-05 01:19:48 INFO  AbstractOperator:380 - Reconciliation #1076(timer) KafkaConnect(kafka-connect/pars-kc): Reconciliation is in progress
2023-06-05 01:19:48 INFO  AbstractOperator:380 - Reconciliation #1079(timer) KafkaConnect(kafka-connect/ccat): Reconciliation is in progress
2023-06-05 01:19:49 ERROR Util:166 - Reconciliation #1076(timer) KafkaConnect(kafka-connect/pars-kc): Exceeded timeout of 300000ms while waiting for Deployment resource pars-kc-connect in namespace kafka-connect to be ready
2023-06-05 01:19:49 ERROR AbstractOperator:260 - Reconciliation #1076(timer) KafkaConnect(kafka-connect/pars-kc): createOrUpdate failed
io.strimzi.operator.common.operator.resource.TimeoutException: Exceeded timeout of 300000ms while waiting for Deployment resource pars-kc-connect in namespace kafka-connect to be ready
        at io.strimzi.operator.common.Util$1.lambda$handle$1(Util.java:167) ~[io.strimzi.operator-common-0.35.0.jar:0.35.0]
        at io.vertx.core.impl.future.FutureImpl$3.onFailure(FutureImpl.java:153) ~[io.vertx.vertx-core-4.3.8.jar:4.3.8]
        at io.vertx.core.impl.future.FutureBase.lambda$emitFailure$1(FutureBase.java:69) ~[io.vertx.vertx-core-4.3.8.jar:4.3.8]
        at io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:174) ~[io.netty.netty-common-4.1.87.Final.jar:4.1.87.Final]
        at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:167) ~[io.netty.netty-common-4.1.87.Final.jar:4.1.87.Final]
        at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:470) ~[io.netty.netty-common-4.1.87.Final.jar:4.1.87.Final]
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:569) ~[io.netty.netty-transport-4.1.87.Final.jar:4.1.87.Final]
        at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) ~[io.netty.netty-common-4.1.87.Final.jar:4.1.87.Final]
        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[io.netty.netty-common-4.1.87.Final.jar:4.1.87.Final]
        at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[io.netty.netty-common-4.1.87.Final.jar:4.1.87.Final]
        at java.lang.Thread.run(Thread.java:833) ~[?:?]
2023-06-05 01:19:49 INFO  CrdOperator:133 - Reconciliation #1076(timer) KafkaConnect(kafka-connect/pars-kc): Status of KafkaConnect pars-kc in namespace kafka-connect has been updated
2023-06-05 01:19:49 WARN  AbstractOperator:525 - Reconciliation #1076(timer) KafkaConnect(kafka-connect/pars-kc): Failed to reconcile
io.strimzi.operator.common.operator.resource.TimeoutException: Exceeded timeout of 300000ms while waiting for Deployment resource pars-kc-connect in namespace kafka-connect to be ready
        at io.strimzi.operator.common.Util$1.lambda$handle$1(Util.java:167) ~[io.strimzi.operator-common-0.35.0.jar:0.35.0]
        at io.vertx.core.impl.future.FutureImpl$3.onFailure(FutureImpl.java:153) ~[io.vertx.vertx-core-4.3.8.jar:4.3.8]
        at io.vertx.core.impl.future.FutureBase.lambda$emitFailure$1(FutureBase.java:69) ~[io.vertx.vertx-core-4.3.8.jar:4.3.8]
        at io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:174) ~[io.netty.netty-common-4.1.87.Final.jar:4.1.87.Final]
        at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:167) ~[io.netty.netty-common-4.1.87.Final.jar:4.1.87.Final]
        at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:470) ~[io.netty.netty-common-4.1.87.Final.jar:4.1.87.Final]
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:569) ~[io.netty.netty-transport-4.1.87.Final.jar:4.1.87.Final]
        at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) ~[io.netty.netty-common-4.1.87.Final.jar:4.1.87.Final]
        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[io.netty.netty-common-4.1.87.Final.jar:4.1.87.Final]
        at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[io.netty.netty-common-4.1.87.Final.jar:4.1.87.Final]
        at java.lang.Thread.run(Thread.java:833) ~[?:?]

imrohankataria Jun 5, 2023
Author

Just got my pods running, wuhoo.
I was playing with adding more nodes, and changed the resource limit

Maybe I was giving a lot before so reducing it to reasonable range did the work? or probably adding another few nodes?
I'd love to know the answer to this. What logically could have happened? and How to put the resource limits?

imrohankataria Jun 5, 2023
Author

Just sharing with this thread what's happening in realtime, so I can update folks here more on the issue -

The restart journey of pods has started again, and I want to understand why it happens, hence how to control it?

pars-kc-connect-6d7bc6684d-jm458            0/1     Running   3 (42s ago)   19m

Events -

108s        Normal    Scheduled           pod/pars-kc-connect-57fdd885cc-npg79    Successfully assigned kafka-connect/pars-kc-connect-57fdd885cc-npg79 to aks-agentpool-38288350-vmss000008
108s        Normal    Pulled              pod/pars-kc-connect-57fdd885cc-npg79    Container image "acrparsdevcentralus.azurecr.io/connect@sha256:843dbf5b64340b48bb0388d3b54c4d071fb10fbe8d3bf96cbfd4afe3210f064c" already present on machine
108s        Normal    Created             pod/pars-kc-connect-57fdd885cc-npg79    Created container pars-kc-connect
108s        Normal    Started             pod/pars-kc-connect-57fdd885cc-npg79    Started container pars-kc-connect
108s        Normal    SuccessfulCreate    replicaset/pars-kc-connect-57fdd885cc   Created pod: pars-kc-connect-57fdd885cc-npg79
54m         Warning   FailedScheduling    pod/pars-kc-connect-5bdfc649c7-6ksl4    0/1 nodes are available: 1 Insufficient cpu, 1 Insufficient memory. preemption: 0/1 nodes are available: 1 No preemption victims found for incoming pod.
49m         Normal    NotTriggerScaleUp   pod/pars-kc-connect-5bdfc649c7-6ksl4    pod didn't trigger scale-up: 1 Insufficient cpu, 1 Insufficient memory
49m         Warning   FailedScheduling    pod/pars-kc-connect-5bdfc649c7-6ksl4    0/1 nodes are available: 1 Insufficient cpu, 1 Insufficient memory. preemption: 0/1 nodes are available: 1 No preemption victims found for incoming pod.
47m         Warning   FailedScheduling    pod/pars-kc-connect-5bdfc649c7-6ksl4    skip schedule deleting pod: kafka-connect/pars-kc-connect-5bdfc649c7-6ksl4
54m         Normal    SuccessfulCreate    replicaset/pars-kc-connect-5bdfc649c7   Created pod: pars-kc-connect-5bdfc649c7-6ksl4
55m         Normal    NotTriggerScaleUp   pod/pars-kc-connect-5fd66767b-xh6b4     pod didn't trigger scale-up: 1 Insufficient cpu, 1 Insufficient memory
50m         Normal    NotTriggerScaleUp   pod/pars-kc-connect-5fd66767b-xh6b4     pod didn't trigger scale-up: 1 Insufficient memory, 1 Insufficient cpu
50m         Warning   FailedScheduling    pod/pars-kc-connect-5fd66767b-xh6b4     0/1 nodes are available: 1 Insufficient cpu, 1 Insufficient memory. preemption: 0/1 nodes are available: 1 No preemption victims found for incoming pod.
47m         Warning   FailedScheduling    pod/pars-kc-connect-5fd66767b-xh6b4     skip schedule deleting pod: kafka-connect/pars-kc-connect-5fd66767b-xh6b4
21m         Warning   FailedScheduling    pod/pars-kc-connect-6d7bc6684d-jm458    0/4 nodes are available: 1 Insufficient cpu, 3 Insufficient memory. preemption: 0/4 nodes are available: 4 No preemption victims found for incoming pod.
21m         Normal    TriggeredScaleUp    pod/pars-kc-connect-6d7bc6684d-jm458    pod triggered scale-up: [{aks-agentpool-38288350-vmss 1->2 (max: 16)}]
20m         Warning   FailedScheduling    pod/pars-kc-connect-6d7bc6684d-jm458    0/5 nodes are available: 1 Insufficient cpu, 1 node(s) had untolerated taint {node.cloudprovider.kubernetes.io/uninitialized: true}, 3 Insufficient memory. preemption: 0/5 nodes are available: 1 Preemption is not helpful for scheduling, 4 No preemption victims found for incoming pod.
20m         Normal    Scheduled           pod/pars-kc-connect-6d7bc6684d-jm458    Successfully assigned kafka-connect/pars-kc-connect-6d7bc6684d-jm458 to aks-agentpool-38288350-vmss000008
20m         Normal    Pulling             pod/pars-kc-connect-6d7bc6684d-jm458    Pulling image "acrparsdevcentralus.azurecr.io/connect@sha256:843dbf5b64340b48bb0388d3b54c4d071fb10fbe8d3bf96cbfd4afe3210f064c"
19m         Normal    Pulled              pod/pars-kc-connect-6d7bc6684d-jm458    Successfully pulled image "acrparsdevcentralus.azurecr.io/connect@sha256:843dbf5b64340b48bb0388d3b54c4d071fb10fbe8d3bf96cbfd4afe3210f064c" in 23.425172592s
3m40s       Normal    Created             pod/pars-kc-connect-6d7bc6684d-jm458    Created container pars-kc-connect
3m40s       Normal    Started             pod/pars-kc-connect-6d7bc6684d-jm458    Started container pars-kc-connect
3m47s       Warning   Unhealthy           pod/pars-kc-connect-6d7bc6684d-jm458    Liveness probe failed: Get "http://10.244.11.4:8083/": dial tcp 10.244.11.4:8083: connect: connection refused
3m47s       Warning   Unhealthy           pod/pars-kc-connect-6d7bc6684d-jm458    Readiness probe failed: Get "http://10.244.11.4:8083/": dial tcp 10.244.11.4:8083: connect: connection refused
6m47s       Normal    Killing             pod/pars-kc-connect-6d7bc6684d-jm458    Container pars-kc-connect failed liveness probe, will be restarted
2m35s       Normal    Pulled              pod/pars-kc-connect-6d7bc6684d-jm458    Container image "acrparsdevcentralus.azurecr.io/connect@sha256:843dbf5b64340b48bb0388d3b54c4d071fb10fbe8d3bf96cbfd4afe3210f064c" already present on machine
2m47s       Warning   BackOff             pod/pars-kc-connect-6d7bc6684d-jm458    Back-off restarting failed container
21m         Normal    SuccessfulCreate    replicaset/pars-kc-connect-6d7bc6684d   Created pod: pars-kc-connect-6d7bc6684d-jm458
48s         Normal    SuccessfulDelete    replicaset/pars-kc-connect-6d7bc6684d   Deleted pod: pars-kc-connect-6d7bc6684d-jm458
56m         Normal    NotTriggerScaleUp   pod/pars-kc-connect-787c55f74f-lzhqx    pod didn't trigger scale-up: 1 Insufficient cpu, 1 Insufficient memory
57m         Warning   FailedScheduling    pod/pars-kc-connect-787c55f74f-lzhqx    0/1 nodes are available: 1 Insufficient cpu, 1 Insufficient memory. preemption: 0/1 nodes are available: 1 No preemption victims found for incoming pod.
54m         Warning   FailedScheduling    pod/pars-kc-connect-787c55f74f-lzhqx    skip schedule deleting pod: kafka-connect/pars-kc-connect-787c55f74f-lzhqx
54m         Normal    SuccessfulDelete    replicaset/pars-kc-connect-787c55f74f   Deleted pod: pars-kc-connect-787c55f74f-lzhqx
38m         Warning   FailedScheduling    pod/pars-kc-connect-85c4bf9d46-mrpk6    0/1 nodes are available: 1 Insufficient cpu, 1 Insufficient memory. preemption: 0/1 nodes are available: 1 No preemption victims found for incoming pod.
38m         Normal    NotTriggerScaleUp   pod/pars-kc-connect-85c4bf9d46-mrpk6    pod didn't trigger scale-up: 1 Insufficient memory, 1 Insufficient cpu
28m         Normal    NotTriggerScaleUp   pod/pars-kc-connect-85c4bf9d46-mrpk6    pod didn't trigger scale-up: 1 Insufficient cpu, 1 Insufficient memory
21m         Warning   FailedScheduling    pod/pars-kc-connect-85c4bf9d46-mrpk6    0/1 nodes are available: 1 Insufficient cpu, 1 Insufficient memory. preemption: 0/1 nodes are available: 1 No preemption victims found for incoming pod.
23m         Normal    NotTriggerScaleUp   pod/pars-kc-connect-85c4bf9d46-mrpk6    pod didn't trigger scale-up: 2 Insufficient cpu, 2 Insufficient memory
20m         Warning   FailedScheduling    pod/pars-kc-connect-85c4bf9d46-mrpk6    0/5 nodes are available: 1 node(s) had untolerated taint {node.cloudprovider.kubernetes.io/uninitialized: true}, 4 Insufficient cpu, 4 Insufficient memory. preemption: 0/5 nodes are available: 1 Preemption is not helpful for scheduling, 4 No preemption victims found for incoming pod.
18m         Warning   FailedScheduling    pod/pars-kc-connect-85c4bf9d46-mrpk6    0/5 nodes are available: 5 Insufficient cpu, 5 Insufficient memory. preemption: 0/5 nodes are available: 5 No preemption victims found for incoming pod.
18m         Warning   FailedScheduling    pod/pars-kc-connect-85c4bf9d46-mrpk6    skip schedule deleting pod: kafka-connect/pars-kc-connect-85c4bf9d46-mrpk6
38m         Normal    SuccessfulCreate    replicaset/pars-kc-connect-85c4bf9d46   Created pod: pars-kc-connect-85c4bf9d46-mrpk6
18m         Normal    SuccessfulDelete    replicaset/pars-kc-connect-85c4bf9d46   Deleted pod: pars-kc-connect-85c4bf9d46-mrpk6
41m         Normal    Scheduled           pod/pars-kc-connect-build               Successfully assigned kafka-connect/pars-kc-connect-build to aks-agentpool-38288350-vmss000000
41m         Normal    Pulled              pod/pars-kc-connect-build               Container image "quay.io/strimzi/kaniko-executor:0.35.0" already present on machine
41m         Normal    Created             pod/pars-kc-connect-build               Created container pars-kc-connect-build
41m         Normal    Started             pod/pars-kc-connect-build               Started container pars-kc-connect-build
54m         Normal    ScalingReplicaSet   deployment/pars-kc-connect              Scaled down replica set pars-kc-connect-787c55f74f to 0
54m         Normal    ScalingReplicaSet   deployment/pars-kc-connect              Scaled up replica set pars-kc-connect-5bdfc649c7 to 1
38m         Normal    NoPods              poddisruptionbudget/pars-kc-connect     No matching pods found
38m         Normal    ScalingReplicaSet   deployment/pars-kc-connect              Scaled up replica set pars-kc-connect-85c4bf9d46 to 1
21m         Normal    ScalingReplicaSet   deployment/pars-kc-connect              Scaled up replica set pars-kc-connect-6d7bc6684d to 1
18m         Normal    ScalingReplicaSet   deployment/pars-kc-connect              Scaled down replica set pars-kc-connect-85c4bf9d46 to 0
108s        Normal    ScalingReplicaSet   deployment/pars-kc-connect              Scaled up replica set pars-kc-connect-57fdd885cc to 1
48s         Normal    ScalingReplicaSet   deployment/pars-kc-connect              Scaled down replica set pars-kc-connect-6d7bc6684d to 0

Operator Logs -

2023-06-05 02:03:04 WARN  AbstractOperator:525 - Reconciliation #1123(timer) KafkaConnect(kafka-connect/ccat): Failed to reconcile
io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: ccat-connect-api.kafka-connect.svc.cluster.local/10.0.187.155:8083
Caused by: java.net.ConnectException: Connection refused
        at sun.nio.ch.Net.pollConnect(Native Method) ~[?:?]
        at sun.nio.ch.Net.pollConnectNow(Net.java:672) ~[?:?]
        at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:946) ~[?:?]
        at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:337) ~[io.netty.netty-transport-4.1.87.Final.jar:4.1.87.Final]
        at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:334) ~[io.netty.netty-transport-4.1.87.Final.jar:4.1.87.Final]
        at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:776) ~[io.netty.netty-transport-4.1.87.Final.jar:4.1.87.Final]
        at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) ~[io.netty.netty-transport-4.1.87.Final.jar:4.1.87.Final]
        at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) ~[io.netty.netty-transport-4.1.87.Final.jar:4.1.87.Final]
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) ~[io.netty.netty-transport-4.1.87.Final.jar:4.1.87.Final]
        at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) ~[io.netty.netty-common-4.1.87.Final.jar:4.1.87.Final]
        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[io.netty.netty-common-4.1.87.Final.jar:4.1.87.Final]
        at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[io.netty.netty-common-4.1.87.Final.jar:4.1.87.Final]
        at java.lang.Thread.run(Thread.java:833) ~[?:?]
2023-06-05 02:03:07 WARN  AbstractConnectOperator:944 - Reconciliation #1122(timer) KafkaConnect(kafka-connect/pars-kc): Error reconciling connector ccat-connector-sink
io.strimzi.operator.cluster.operator.assembly.ConnectRestException: PUT /connectors/ccat-connector-sink/config returned 400 (Bad Request): Connector configuration is invalid and contains the following 1 error(s):
Invalid value com.snowflake.kafka.connector.records.SnowflakeJsonConvert for configuration value.converter: Class com.snowflake.kafka.connector.records.SnowflakeJsonConvert could not be found.
You can also find the above list of errors at the endpoint `/connector-plugins/{connectorType}/config/validate`
        at io.strimzi.operator.cluster.operator.assembly.KafkaConnectApiImpl.lambda$createOrUpdatePutRequest$1(KafkaConnectApiImpl.java:92) ~[io.strimzi.cluster-operator-0.35.0.jar:0.35.0]
        at io.vertx.core.impl.future.FutureImpl$1.onSuccess(FutureImpl.java:91) ~[io.vertx.vertx-core-4.3.8.jar:4.3.8]
        at io.vertx.core.impl.future.FutureBase.lambda$emitSuccess$0(FutureBase.java:54) ~[io.vertx.vertx-core-4.3.8.jar:4.3.8]
        at io.vertx.core.impl.EventLoopContext.execute(EventLoopContext.java:86) ~[io.vertx.vertx-core-4.3.8.jar:4.3.8]
        at io.vertx.core.impl.ContextBase.execute(ContextBase.java:225) ~[io.vertx.vertx-core-4.3.8.jar:4.3.8]
        at io.vertx.core.impl.future.FutureBase.emitSuccess(FutureBase.java:51) ~[io.vertx.vertx-core-4.3.8.jar:4.3.8]
        at io.vertx.core.impl.future.FutureImpl.tryComplete(FutureImpl.java:211) ~[io.vertx.vertx-core-4.3.8.jar:4.3.8]
        at io.vertx.core.impl.future.PromiseImpl.tryComplete(PromiseImpl.java:23) ~[io.vertx.vertx-core-4.3.8.jar:4.3.8]
        at io.vertx.core.http.impl.HttpEventHandler.handleEnd(HttpEventHandler.java:79) ~[io.vertx.vertx-core-4.3.8.jar:4.3.8]
        at io.vertx.core.http.impl.HttpClientResponseImpl.handleEnd(HttpClientResponseImpl.java:250) ~[io.vertx.vertx-core-4.3.8.jar:4.3.8]
        at io.vertx.core.http.impl.Http1xClientConnection$StreamImpl.lambda$new$0(Http1xClientConnection.java:435) ~[io.vertx.vertx-core-4.3.8.jar:4.3.8]
        at io.vertx.core.streams.impl.InboundBuffer.handleEvent(InboundBuffer.java:239) ~[io.vertx.vertx-core-4.3.8.jar:4.3.8]
        at io.vertx.core.streams.impl.InboundBuffer.write(InboundBuffer.java:129) ~[io.vertx.vertx-core-4.3.8.jar:4.3.8]
        at io.vertx.core.http.impl.Http1xClientConnection$StreamImpl.handleEnd(Http1xClientConnection.java:682) ~[io.vertx.vertx-core-4.3.8.jar:4.3.8]
        at io.vertx.core.impl.EventLoopContext.execute(EventLoopContext.java:76) ~[io.vertx.vertx-core-4.3.8.jar:4.3.8]
        at io.vertx.core.impl.ContextBase.execute(ContextBase.java:232) ~[io.vertx.vertx-core-4.3.8.jar:4.3.8]
        at io.vertx.core.http.impl.Http1xClientConnection.handleResponseEnd(Http1xClientConnection.java:916) ~[io.vertx.vertx-core-4.3.8.jar:4.3.8]
        at io.vertx.core.http.impl.Http1xClientConnection.handleHttpMessage(Http1xClientConnection.java:788) ~[io.vertx.vertx-core-4.3.8.jar:4.3.8]
        at io.vertx.core.http.impl.Http1xClientConnection.handleMessage(Http1xClientConnection.java:752) ~[io.vertx.vertx-core-4.3.8.jar:4.3.8]
        at io.vertx.core.net.impl.ConnectionBase.read(ConnectionBase.java:157) ~[io.vertx.vertx-core-4.3.8.jar:4.3.8]
        at io.vertx.core.net.impl.VertxHandler.channelRead(VertxHandler.java:153) ~[io.vertx.vertx-core-4.3.8.jar:4.3.8]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) ~[io.netty.netty-transport-4.1.87.Final.jar:4.1.87.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[io.netty.netty-transport-4.1.87.Final.jar:4.1.87.Final]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[io.netty.netty-transport-4.1.87.Final.jar:4.1.87.Final]
        at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:436) ~[io.netty.netty-transport-4.1.87.Final.jar:4.1.87.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) ~[io.netty.netty-codec-4.1.87.Final.jar:4.1.87.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) ~[io.netty.netty-codec-4.1.87.Final.jar:4.1.87.Final]
        at io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:251) ~[io.netty.netty-transport-4.1.87.Final.jar:4.1.87.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) ~[io.netty.netty-transport-4.1.87.Final.jar:4.1.87.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[io.netty.netty-transport-4.1.87.Final.jar:4.1.87.Final]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[io.netty.netty-transport-4.1.87.Final.jar:4.1.87.Final]
        at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:280) ~[io.netty.netty-handler-4.1.87.Final.jar:4.1.87.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) ~[io.netty.netty-transport-4.1.87.Final.jar:4.1.87.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[io.netty.netty-transport-4.1.87.Final.jar:4.1.87.Final]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[io.netty.netty-transport-4.1.87.Final.jar:4.1.87.Final]
        at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[io.netty.netty-transport-4.1.87.Final.jar:4.1.87.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) ~[io.netty.netty-transport-4.1.87.Final.jar:4.1.87.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[io.netty.netty-transport-4.1.87.Final.jar:4.1.87.Final]
        at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[io.netty.netty-transport-4.1.87.Final.jar:4.1.87.Final]
        at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) ~[io.netty.netty-transport-4.1.87.Final.jar:4.1.87.Final]
        at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) ~[io.netty.netty-transport-4.1.87.Final.jar:4.1.87.Final]
        at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) ~[io.netty.netty-transport-4.1.87.Final.jar:4.1.87.Final]
        at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) ~[io.netty.netty-transport-4.1.87.Final.jar:4.1.87.Final]
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) ~[io.netty.netty-transport-4.1.87.Final.jar:4.1.87.Final]
        at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) ~[io.netty.netty-common-4.1.87.Final.jar:4.1.87.Final]
        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[io.netty.netty-common-4.1.87.Final.jar:4.1.87.Final]
        at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[io.netty.netty-common-4.1.87.Final.jar:4.1.87.Final]
        at java.lang.Thread.run(Thread.java:833) ~[?:?]
2023-06-05 02:03:07 INFO  CrdOperator:133 - Reconciliation #1122(timer) KafkaConnect(kafka-connect/pars-kc): Status of KafkaConnector ccat-connector-sink in namespace kafka-connect has been updated
2023-06-05 02:03:48 INFO  AbstractOperator:380 - Reconciliation #1122(timer) KafkaConnect(kafka-connect/pars-kc): Reconciliation is in progress
2023-06-05 02:04:05 ERROR AbstractOperator:492 - Watcher closed with exception in namespace kafka-connect
io.fabric8.kubernetes.client.WatcherException: Could not process websocket message
        at io.fabric8.kubernetes.client.dsl.internal.WatcherWebSocketListener.onError(WatcherWebSocketListener.java:51) ~[io.fabric8.kubernetes-client-6.6.1.jar:?]
        at io.fabric8.kubernetes.client.jdkhttp.JdkWebSocketImpl$ListenerAdapter.onError(JdkWebSocketImpl.java:85) ~[io.fabric8.kubernetes-httpclient-jdk-6.6.1.jar:?]
        at jdk.internal.net.http.websocket.WebSocketImpl$ReceiveTask.processError(WebSocketImpl.java:508) ~[java.net.http:?]
        at jdk.internal.net.http.websocket.WebSocketImpl$ReceiveTask.run(WebSocketImpl.java:462) ~[java.net.http:?]
        at jdk.internal.net.http.common.SequentialScheduler$CompleteRestartableTask.run(SequentialScheduler.java:149) ~[java.net.http:?]
        at jdk.internal.net.http.common.SequentialScheduler$SchedulableTask.run(SequentialScheduler.java:230) ~[java.net.http:?]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?]
        at java.lang.Thread.run(Thread.java:833) ~[?:?]
Caused by: java.net.SocketException: Connection reset
        at sun.nio.ch.SocketChannelImpl.throwConnectionReset(SocketChannelImpl.java:394) ~[?:?]
        at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:426) ~[?:?]
        at jdk.internal.net.http.SocketTube.readAvailable(SocketTube.java:1170) ~[java.net.http:?]
        at jdk.internal.net.http.SocketTube$InternalReadPublisher$InternalReadSubscription.read(SocketTube.java:833) ~[java.net.http:?]
        at jdk.internal.net.http.SocketTube$SocketFlowTask.run(SocketTube.java:181) ~[java.net.http:?]
        at jdk.internal.net.http.common.SequentialScheduler$SchedulableTask.run(SequentialScheduler.java:230) ~[java.net.http:?]
        at jdk.internal.net.http.common.SequentialScheduler.runOrSchedule(SequentialScheduler.java:303) ~[java.net.http:?]
        at jdk.internal.net.http.common.SequentialScheduler.runOrSchedule(SequentialScheduler.java:256) ~[java.net.http:?]
        at jdk.internal.net.http.SocketTube$InternalReadPublisher$InternalReadSubscription.signalReadable(SocketTube.java:774) ~[java.net.http:?]
        at jdk.internal.net.http.SocketTube$InternalReadPublisher$ReadEvent.signalEvent(SocketTube.java:957) ~[java.net.http:?]
        at jdk.internal.net.http.SocketTube$SocketFlowEvent.handle(SocketTube.java:253) ~[java.net.http:?]
        at jdk.internal.net.http.HttpClientImpl$SelectorManager.handleEvent(HttpClientImpl.java:979) ~[java.net.http:?]
        at jdk.internal.net.http.HttpClientImpl$SelectorManager.lambda$run$3(HttpClientImpl.java:934) ~[java.net.http:?]
        at java.util.ArrayList.forEach(ArrayList.java:1511) ~[?:?]
        at jdk.internal.net.http.HttpClientImpl$SelectorManager.run(HttpClientImpl.java:934) ~[java.net.http:?]

scholzj Jun 5, 2023
Maintainer

Your Pod is pending because you don't have resources for it: 0/1 nodes are available: 1 Insufficient cpu, 1 Insufficient memory. preemption: 0/1 nodes are available: 1 No preemption victims found for incoming pod.. You need to make sure your cluster has enough resources to accommodate the Pod which does not seem to be the case.

Setting a different memory request and limit is also quite a bad practice for something like Java. You should ideally use the same value for both.

You did not share any logs from the Kafka Connect pod when it is running. So it is not clear what is going on there.

Answer selected by imrohankataria

imrohankataria Jun 5, 2023
Author

Hello @scholzj thanks for looking into this.
I have made the change as suggested "keeping request and limit resource same"

Below are the logs from a POD that kept on restarting -

Preparing truststore
Preparing truststore is complete
Starting Kafka Connect with configuration:
# Bootstrap servers
bootstrap.servers=evh-pars-dev-centralus.servicebus.windows.net:9093
# REST Listeners
rest.port=8083
rest.advertised.host.name=10.244.0.33
rest.advertised.port=8083
# Plugins
plugin.path=/opt/kafka/plugins
# Provided configuration
offset.storage.topic=strimzi-connect-cluster-offsets
value.converter=org.apache.kafka.connect.json.JsonConverter
config.storage.topic=strimzi-connect-cluster-configs
key.converter=org.apache.kafka.connect.json.JsonConverter
group.id=connect-cluster
status.storage.topic=strimzi-connect-cluster-status
group.Id=123
kafka.heap.opts=-Xms1G -Xmx2G
key.converter.schemas.enable=false
offset.flush.internal.ms=10000
request.timeout.ms=60000
value.converter.schemas.enable=true


security.protocol=SASL_SSL
producer.security.protocol=SASL_SSL
consumer.security.protocol=SASL_SSL
admin.security.protocol=SASL_SSL


sasl.mechanism=PLAIN
sasl.jaas.config=[hidden]

producer.sasl.mechanism=PLAIN
producer.sasl.jaas.config=[hidden]

consumer.sasl.mechanism=PLAIN
consumer.sasl.jaas.config=[hidden]

admin.sasl.mechanism=PLAIN
admin.sasl.jaas.config=[hidden]

# Additional configuration
consumer.client.rack=

Configuring Java heap: -Xms1556925644 -Xmx1556925644
+ exec /usr/bin/tini -w -e 143 -- /opt/kafka/bin/connect-distributed.sh /tmp/strimzi-connect.properties
2023-06-05 13:43:53,321 INFO WorkerInfo values: 
	jvm.args = -Xms1556925644, -Xmx1556925644, -XX:+UseG1GC, -XX:MaxGCPauseMillis=20, -XX:InitiatingHeapOccupancyPercent=35, -XX:+ExplicitGCInvokesConcurrent, -XX:MaxInlineLevel=15, -Djava.awt.headless=true, -Dcom.sun.management.jmxremote, -Dcom.sun.management.jmxremote.authenticate=false, -Dcom.sun.management.jmxremote.ssl=false, -Dkafka.logs.dir=/opt/kafka, -Dlog4j.configuration=file:/opt/kafka/custom-config/log4j.properties
	jvm.spec = Red Hat, Inc., OpenJDK 64-Bit Server VM, 17.0.7, 17.0.7+7-LTS
	jvm.classpath = /opt/kafka/bin/../libs/accessors-smart-2.4.9.jar:/opt/kafka/bin/../libs/activation-1.1.1.jar:/opt/kafka/bin/../libs/annotations-13.0.jar:/opt/kafka/bin/../libs/aopalliance-repackaged-2.6.1.jar:/opt/kafka/bin/../libs/argparse4j-0.7.0.jar:/opt/kafka/bin/../libs/audience-annotations-0.5.0.jar:/opt/kafka/bin/../libs/checker-qual-3.5.0.jar:/opt/kafka/bin/../libs/commons-cli-1.4.jar:/opt/kafka/bin/../libs/commons-lang-2.6.jar:/opt/kafka/bin/../libs/commons-lang3-3.8.1.jar:/opt/kafka/bin/../libs/connect-api-3.4.0.jar:/opt/kafka/bin/../libs/connect-basic-auth-extension-3.4.0.jar:/opt/kafka/bin/../libs/connect-json-3.4.0.jar:/opt/kafka/bin/../libs/connect-mirror-3.4.0.jar:/opt/kafka/bin/../libs/connect-mirror-client-3.4.0.jar:/opt/kafka/bin/../libs/connect-runtime-3.4.0.jar:/opt/kafka/bin/../libs/connect-transforms-3.4.0.jar:/opt/kafka/bin/../libs/cruise-control-metrics-reporter-2.5.112.jar:/opt/kafka/bin/../libs/error_prone_annotations-2.3.4.jar:/opt/kafka/bin/../libs/failureaccess-1.0.1.jar:/opt/kafka/bin/../libs/gson-2.9.0.jar:/opt/kafka/bin/../libs/guava-30.1-jre.jar:/opt/kafka/bin/../libs/hk2-api-2.6.1.jar:/opt/kafka/bin/../libs/hk2-locator-2.6.1.jar:/opt/kafka/bin/../libs/hk2-utils-2.6.1.jar:/opt/kafka/bin/../libs/j2objc-annotations-1.3.jar:/opt/kafka/bin/../libs/jackson-annotations-2.13.4.jar:/opt/kafka/bin/../libs/jackson-core-2.13.4.jar:/opt/kafka/bin/../libs/jackson-databind-2.13.4.2.jar:/opt/kafka/bin/../libs/jackson-dataformat-csv-2.13.4.jar:/opt/kafka/bin/../libs/jackson-dataformat-yaml-2.14.2.jar:/opt/kafka/bin/../libs/jackson-datatype-jdk8-2.13.4.jar:/opt/kafka/bin/../libs/jackson-datatype-jsr310-2.14.2.jar:/opt/kafka/bin/../libs/jackson-jaxrs-base-2.13.4.jar:/opt/kafka/bin/../libs/jackson-jaxrs-json-provider-2.13.4.jar:/opt/kafka/bin/../libs/jackson-module-jaxb-annotations-2.13.4.jar:/opt/kafka/bin/../libs/jackson-module-scala_2.13-2.13.4.jar:/opt/kafka/bin/../libs/jaeger-client-1.8.1.jar:/opt/kafka/bin/../libs/jaeger-core-1.8.1.jar:/opt/kafka/bin/../libs/jaeger-thrift-1.8.1.jar:/opt/kafka/bin/../libs/jaeger-tracerresolver-1.8.1.jar:/opt/kafka/bin/../libs/jakarta.activation-api-1.2.2.jar:/opt/kafka/bin/../libs/jakarta.annotation-api-1.3.5.jar:/opt/kafka/bin/../libs/jakarta.inject-2.6.1.jar:/opt/kafka/bin/../libs/jakarta.validation-api-2.0.2.jar:/opt/kafka/bin/../libs/jakarta.ws.rs-api-2.1.6.jar:/opt/kafka/bin/../libs/jakarta.xml.bind-api-2.3.3.jar:/opt/kafka/bin/../libs/javassist-3.27.0-GA.jar:/opt/kafka/bin/../libs/javax.annotation-api-1.3.2.jar:/opt/kafka/bin/../libs/javax.servlet-api-3.1.0.jar:/opt/kafka/bin/../libs/javax.ws.rs-api-2.1.1.jar:/opt/kafka/bin/../libs/jaxb-api-2.3.0.jar:/opt/kafka/bin/../libs/jersey-client-2.34.jar:/opt/kafka/bin/../libs/jersey-common-2.34.jar:/opt/kafka/bin/../libs/jersey-container-servlet-2.34.jar:/opt/kafka/bin/../libs/jersey-container-servlet-core-2.34.jar:/opt/kafka/bin/../libs/jersey-hk2-2.34.jar:/opt/kafka/bin/../libs/jersey-server-2.34.jar:/opt/kafka/bin/../libs/jetty-client-9.4.48.v20220622.jar:/opt/kafka/bin/../libs/jetty-continuation-9.4.48.v20220622.jar:/opt/kafka/bin/../libs/jetty-http-9.4.48.v20220622.jar:/opt/kafka/bin/../libs/jetty-io-9.4.48.v20220622.jar:/opt/kafka/bin/../libs/jetty-security-9.4.48.v20220622.jar:/opt/kafka/bin/../libs/jetty-server-9.4.48.v20220622.jar:/opt/kafka/bin/../libs/jetty-servlet-9.4.48.v20220622.jar:/opt/kafka/bin/../libs/jetty-servlets-9.4.48.v20220622.jar:/opt/kafka/bin/../libs/jetty-util-9.4.48.v20220622.jar:/opt/kafka/bin/../libs/jetty-util-ajax-9.4.48.v20220622.jar:/opt/kafka/bin/../libs/jline-3.21.0.jar:/opt/kafka/bin/../libs/jmx_prometheus_javaagent-0.18.0.jar:/opt/kafka/bin/../libs/jopt-simple-5.0.4.jar:/opt/kafka/bin/../libs/jose4j-0.7.9.jar:/opt/kafka/bin/../libs/json-path-2.6.0.jar:/opt/kafka/bin/../libs/json-smart-2.4.9.jar:/opt/kafka/bin/../libs/jsonevent-layout-1.7.jar:/opt/kafka/bin/../libs/jsr305-3.0.2.jar:/opt/kafka/bin/../libs/kafka-agent-0.35.0.jar:/opt/kafka/bin/../libs/kafka-clients-3.4.0.jar:/opt/kafka/bin/../libs/kafka-env-var-config-provider-1.1.0.jar:/opt/kafka/bin/../libs/kafka-group-coordinator-3.4.0.jar:/opt/kafka/bin/../libs/kafka-kubernetes-config-provider-1.1.1.jar:/opt/kafka/bin/../libs/kafka-log4j-appender-3.4.0.jar:/opt/kafka/bin/../libs/kafka-metadata-3.4.0.jar:/opt/kafka/bin/../libs/kafka-oauth-client-0.12.0.jar:/opt/kafka/bin/../libs/kafka-oauth-common-0.12.0.jar:/opt/kafka/bin/../libs/kafka-oauth-keycloak-authorizer-0.12.0.jar:/opt/kafka/bin/../libs/kafka-oauth-server-0.12.0.jar:/opt/kafka/bin/../libs/kafka-oauth-server-plain-0.12.0.jar:/opt/kafka/bin/../libs/kafka-quotas-plugin-0.2.0.jar:/opt/kafka/bin/../libs/kafka-raft-3.4.0.jar:/opt/kafka/bin/../libs/kafka-server-common-3.4.0.jar:/opt/kafka/bin/../libs/kafka-shell-3.4.0.jar:/opt/kafka/bin/../libs/kafka-storage-3.4.0.jar:/opt/kafka/bin/../libs/kafka-storage-api-3.4.0.jar:/opt/kafka/bin/../libs/kafka-streams-3.4.0.jar:/opt/kafka/bin/../libs/kafka-streams-examples-3.4.0.jar:/opt/kafka/bin/../libs/kafka-streams-scala_2.13-3.4.0.jar:/opt/kafka/bin/../libs/kafka-streams-test-utils-3.4.0.jar:/opt/kafka/bin/../libs/kafka-tools-3.4.0.jar:/opt/kafka/bin/../libs/kafka_2.13-3.4.0.jar:/opt/kafka/bin/../libs/kotlin-stdlib-1.4.10.jar:/opt/kafka/bin/../libs/kotlin-stdlib-common-1.4.0.jar:/opt/kafka/bin/../libs/kubernetes-client-6.5.0.jar:/opt/kafka/bin/../libs/kubernetes-client-api-6.5.0.jar:/opt/kafka/bin/../libs/kubernetes-httpclient-jdk-6.5.0.jar:/opt/kafka/bin/../libs/kubernetes-model-apps-6.5.0.jar:/opt/kafka/bin/../libs/kubernetes-model-autoscaling-6.5.0.jar:/opt/kafka/bin/../libs/kubernetes-model-batch-6.5.0.jar:/opt/kafka/bin/../libs/kubernetes-model-certificates-6.5.0.jar:/opt/kafka/bin/../libs/kubernetes-model-common-6.5.0.jar:/opt/kafka/bin/../libs/kubernetes-model-core-6.5.0.jar:/opt/kafka/bin/../libs/kubernetes-model-extensions-6.5.0.jar:/opt/kafka/bin/../libs/kubernetes-model-gatewayapi-6.5.0.jar:/opt/kafka/bin/../libs/kubernetes-model-resource-6.5.0.jar:/opt/kafka/bin/../libs/libthrift-0.15.0.jar:/opt/kafka/bin/../libs/listenablefuture-9999.0-empty-to-avoid-conflict-with-guava.jar:/opt/kafka/bin/../libs/lz4-java-1.8.0.jar:/opt/kafka/bin/../libs/maven-artifact-3.8.4.jar:/opt/kafka/bin/../libs/metrics-core-2.2.0.jar:/opt/kafka/bin/../libs/metrics-core-4.1.12.1.jar:/opt/kafka/bin/../libs/mirror-maker-2-extensions-1.2.0.jar:/opt/kafka/bin/../libs/mirror-maker-agent-0.35.0.jar:/opt/kafka/bin/../libs/netty-buffer-4.1.78.Final.jar:/opt/kafka/bin/../libs/netty-codec-4.1.78.Final.jar:/opt/kafka/bin/../libs/netty-common-4.1.78.Final.jar:/opt/kafka/bin/../libs/netty-handler-4.1.78.Final.jar:/opt/kafka/bin/../libs/netty-resolver-4.1.78.Final.jar:/opt/kafka/bin/../libs/netty-transport-4.1.78.Final.jar:/opt/kafka/bin/../libs/netty-transport-classes-epoll-4.1.78.Final.jar:/opt/kafka/bin/../libs/netty-transport-native-epoll-4.1.78.Final.jar:/opt/kafka/bin/../libs/netty-transport-native-unix-common-4.1.78.Final.jar:/opt/kafka/bin/../libs/nimbus-jose-jwt-9.10.jar:/opt/kafka/bin/../libs/okhttp-4.9.3.jar:/opt/kafka/bin/../libs/okio-2.8.0.jar:/opt/kafka/bin/../libs/opa-authorizer-1.5.1.jar:/opt/kafka/bin/../libs/opentelemetry-api-1.18.0.jar:/opt/kafka/bin/../libs/opentelemetry-context-1.18.0.jar:/opt/kafka/bin/../libs/opentelemetry-exporter-common-1.18.0.jar:/opt/kafka/bin/../libs/opentelemetry-exporter-otlp-1.18.0.jar:/opt/kafka/bin/../libs/opentelemetry-exporter-otlp-common-1.18.0.jar:/opt/kafka/bin/../libs/opentelemetry-instrumentation-api-1.18.0.jar:/opt/kafka/bin/../libs/opentelemetry-instrumentation-api-semconv-1.18.0-alpha.jar:/opt/kafka/bin/../libs/opentelemetry-kafka-clients-2.6-1.18.0-alpha.jar:/opt/kafka/bin/../libs/opentelemetry-kafka-clients-common-1.18.0-alpha.jar:/opt/kafka/bin/../libs/opentelemetry-opentracing-shim-1.18.0-alpha.jar:/opt/kafka/bin/../libs/opentelemetry-sdk-1.18.0.jar:/opt/kafka/bin/../libs/opentelemetry-sdk-common-1.18.0.jar:/opt/kafka/bin/../libs/opentelemetry-sdk-extension-autoconfigure-1.18.0-alpha.jar:/opt/kafka/bin/../libs/opentelemetry-sdk-extension-autoconfigure-spi-1.18.0.jar:/opt/kafka/bin/../libs/opentelemetry-sdk-logs-1.18.0-alpha.jar:/opt/kafka/bin/../libs/opentelemetry-sdk-metrics-1.18.0.jar:/opt/kafka/bin/../libs/opentelemetry-sdk-trace-1.18.0.jar:/opt/kafka/bin/../libs/opentelemetry-semconv-1.18.0-alpha.jar:/opt/kafka/bin/../libs/opentracing-api-0.33.0.jar:/opt/kafka/bin/../libs/opentracing-kafka-client-0.1.15.jar:/opt/kafka/bin/../libs/opentracing-noop-0.33.0.jar:/opt/kafka/bin/../libs/opentracing-tracerresolver-0.1.8.jar:/opt/kafka/bin/../libs/opentracing-util-0.33.0.jar:/opt/kafka/bin/../libs/osgi-resource-locator-1.0.3.jar:/opt/kafka/bin/../libs/paranamer-2.8.jar:/opt/kafka/bin/../libs/plexus-utils-3.3.0.jar:/opt/kafka/bin/../libs/reflections-0.9.12.jar:/opt/kafka/bin/../libs/reload4j-1.2.19.jar:/opt/kafka/bin/../libs/rocksdbjni-7.1.2.jar:/opt/kafka/bin/../libs/scala-collection-compat_2.13-2.6.0.jar:/opt/kafka/bin/../libs/scala-java8-compat_2.13-1.0.2.jar:/opt/kafka/bin/../libs/scala-library-2.13.10.jar:/opt/kafka/bin/../libs/scala-logging_2.13-3.9.4.jar:/opt/kafka/bin/../libs/scala-reflect-2.13.10.jar:/opt/kafka/bin/../libs/slf4j-api-1.7.36.jar:/opt/kafka/bin/../libs/slf4j-reload4j-1.7.36.jar:/opt/kafka/bin/../libs/snakeyaml-2.0.jar:/opt/kafka/bin/../libs/snakeyaml-engine-2.6.jar:/opt/kafka/bin/../libs/snappy-java-1.1.8.4.jar:/opt/kafka/bin/../libs/swagger-annotations-2.2.0.jar:/opt/kafka/bin/../libs/tracing-agent-0.35.0.jar:/opt/kafka/bin/../libs/trogdor-3.4.0.jar:/opt/kafka/bin/../libs/zjsonpatch-0.3.0.jar:/opt/kafka/bin/../libs/zookeeper-3.6.3.jar:/opt/kafka/bin/../libs/zookeeper-jute-3.6.3.jar:/opt/kafka/bin/../libs/zstd-jni-1.5.2-1.jar
	os.spec = Linux, amd64, 5.4.0-1105-azure
	os.vcpus = 1
 (org.apache.kafka.connect.runtime.WorkerInfo) [main]
2023-06-05 13:43:53,325 INFO Scanning for plugin classes. This might take a moment ... (org.apache.kafka.connect.cli.ConnectDistributed) [main]
2023-06-05 13:43:53,416 INFO Loading plugin from: /opt/kafka/plugins/debezium-sql-connector (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2023-06-05 13:43:55,720 INFO Registered loader: PluginClassLoader{pluginLocation=file:/opt/kafka/plugins/debezium-sql-connector/} (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2023-06-05 13:43:55,720 INFO Added plugin 'io.debezium.connector.sqlserver.SqlServerConnector' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2023-06-05 13:43:55,720 INFO Added plugin 'io.debezium.converters.ByteArrayConverter' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2023-06-05 13:43:55,720 INFO Added plugin 'io.debezium.converters.BinaryDataConverter' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2023-06-05 13:43:55,720 INFO Added plugin 'io.debezium.converters.CloudEventsConverter' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2023-06-05 13:43:55,721 INFO Added plugin 'io.debezium.transforms.ExtractNewRecordState' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2023-06-05 13:43:55,721 INFO Added plugin 'io.debezium.transforms.outbox.EventRouter' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2023-06-05 13:43:55,721 INFO Added plugin 'io.debezium.transforms.ExtractChangedRecordState' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2023-06-05 13:43:55,721 INFO Added plugin 'io.debezium.transforms.HeaderToValue' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2023-06-05 13:43:55,721 INFO Added plugin 'io.debezium.transforms.partitions.PartitionRouting' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2023-06-05 13:43:55,721 INFO Added plugin 'io.debezium.transforms.partitions.ComputePartition' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2023-06-05 13:43:55,721 INFO Added plugin 'io.debezium.transforms.ByLogicalTableRouter' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2023-06-05 13:43:55,721 INFO Added plugin 'io.debezium.transforms.tracing.ActivateTracingSpan' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2023-06-05 13:43:55,721 INFO Added plugin 'org.apache.kafka.connect.connector.policy.AllConnectorClientConfigOverridePolicy' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2023-06-05 13:43:55,721 INFO Added plugin 'org.apache.kafka.connect.connector.policy.PrincipalConnectorClientConfigOverridePolicy' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2023-06-05 13:43:55,721 INFO Added plugin 'org.apache.kafka.connect.connector.policy.NoneConnectorClientConfigOverridePolicy' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2023-06-05 13:43:55,915 INFO Loading plugin from: /opt/kafka/plugins/kafka-connect-jdbc (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2023-06-05 13:44:05,817 INFO Registered loader: PluginClassLoader{pluginLocation=file:/opt/kafka/plugins/kafka-connect-jdbc/} (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2023-06-05 13:44:05,817 INFO Added plugin 'io.confluent.connect.jdbc.JdbcSinkConnector' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2023-06-05 13:44:05,817 INFO Added plugin 'io.confluent.connect.jdbc.JdbcSourceConnector' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2023-06-05 13:44:05,919 INFO Loading plugin from: /opt/kafka/plugins/bcpkix-fips (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2023-06-05 13:44:07,624 INFO Registered loader: PluginClassLoader{pluginLocation=file:/opt/kafka/plugins/bcpkix-fips/} (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2023-06-05 13:44:07,625 INFO Loading plugin from: /opt/kafka/plugins/bc-fips (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2023-06-05 13:44:08,816 INFO Registered loader: PluginClassLoader{pluginLocation=file:/opt/kafka/plugins/bc-fips/} (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2023-06-05 13:44:08,816 INFO Loading plugin from: /opt/kafka/plugins/snowflake-kafka-connector (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2023-06-05 13:44:36,514 INFO [SF_KAFKA_CONNECTOR] Created KCLogger for class: 'com.snowflake.kafka.connector.Utils' without a Kafka Connect global instance id. (com.snowflake.kafka.connector.internal.KCLogger) [main]
2023-06-05 13:44:36,514 INFO [SF_KAFKA_CONNECTOR] Created KCLogger for class: 'com.snowflake.kafka.connector.SnowflakeSinkConnector' without a Kafka Connect global instance id. (com.snowflake.kafka.connector.internal.KCLogger) [main]
2023-06-05 13:44:36,521 INFO Registered loader: PluginClassLoader{pluginLocation=file:/opt/kafka/plugins/snowflake-kafka-connector/} (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2023-06-05 13:44:36,521 INFO Added plugin 'com.snowflake.kafka.connector.SnowflakeSinkConnector' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2023-06-05 13:44:36,521 INFO Added plugin 'com.snowflake.kafka.connector.records.SnowflakeAvroConverter' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2023-06-05 13:44:36,521 INFO Added plugin 'io.confluent.connect.avro.AvroConverter' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2023-06-05 13:44:36,521 INFO Added plugin 'com.snowflake.kafka.connector.records.SnowflakeAvroConverterWithoutSchemaRegistry' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2023-06-05 13:44:36,521 INFO Added plugin 'org.apache.kafka.connect.json.JsonConverter' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2023-06-05 13:44:36,522 INFO Added plugin 'org.apache.kafka.connect.storage.StringConverter' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2023-06-05 13:44:36,522 INFO Added plugin 'com.snowflake.kafka.connector.records.SnowflakeJsonConverter' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2023-06-05 13:44:36,522 INFO Added plugin 'org.apache.kafka.connect.storage.SimpleHeaderConverter' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2023-06-05 13:44:36,522 INFO Added plugin 'org.apache.kafka.common.config.provider.FileConfigProvider' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2023-06-05 13:44:36,522 INFO Added plugin 'org.apache.kafka.common.config.provider.DirectoryConfigProvider' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2023-06-05 13:44:45,512 INFO Registered loader: jdk.internal.loader.ClassLoaders$AppClassLoader@7ad041f3 (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2023-06-05 13:44:45,512 INFO Added plugin 'org.apache.kafka.connect.tools.VerifiableSinkConnector' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2023-06-05 13:44:45,513 INFO Added plugin 'org.apache.kafka.connect.tools.MockSinkConnector' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2023-06-05 13:44:45,513 INFO Added plugin 'org.apache.kafka.connect.tools.VerifiableSourceConnector' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2023-06-05 13:44:45,513 INFO Added plugin 'org.apache.kafka.connect.mirror.MirrorSourceConnector' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2023-06-05 13:44:45,513 INFO Added plugin 'org.apache.kafka.connect.mirror.MirrorHeartbeatConnector' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2023-06-05 13:44:45,513 INFO Added plugin 'org.apache.kafka.connect.tools.SchemaSourceConnector' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2023-06-05 13:44:45,513 INFO Added plugin 'org.apache.kafka.connect.tools.MockSourceConnector' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2023-06-05 13:44:45,513 INFO Added plugin 'org.apache.kafka.connect.mirror.MirrorCheckpointConnector' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2023-06-05 13:44:45,513 INFO Added plugin 'org.apache.kafka.connect.converters.FloatConverter' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2023-06-05 13:44:45,513 INFO Added plugin 'org.apache.kafka.connect.converters.DoubleConverter' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2023-06-05 13:44:45,513 INFO Added plugin 'org.apache.kafka.connect.converters.ByteArrayConverter' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2023-06-05 13:44:45,513 INFO Added plugin 'org.apache.kafka.connect.converters.LongConverter' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2023-06-05 13:44:45,513 INFO Added plugin 'org.apache.kafka.connect.converters.IntegerConverter' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2023-06-05 13:44:45,513 INFO Added plugin 'org.apache.kafka.connect.converters.ShortConverter' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2023-06-05 13:44:45,513 INFO Added plugin 'org.apache.kafka.connect.transforms.ReplaceField$Value' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2023-06-05 13:44:45,513 INFO Added plugin 'org.apache.kafka.connect.transforms.SetSchemaMetadata$Value' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2023-06-05 13:44:45,513 INFO Added plugin 'org.apache.kafka.connect.transforms.ReplaceField$Key' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2023-06-05 13:44:45,513 INFO Added plugin 'org.apache.kafka.connect.transforms.Filter' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2023-06-05 13:44:45,513 INFO Added plugin 'org.apache.kafka.connect.transforms.InsertField$Value' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2023-06-05 13:44:45,513 INFO Added plugin 'org.apache.kafka.connect.transforms.HeaderFrom$Key' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2023-06-05 13:44:45,514 INFO Added plugin 'org.apache.kafka.connect.transforms.TimestampConverter$Key' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2023-06-05 13:44:45,514 INFO Added plugin 'org.apache.kafka.connect.transforms.MaskField$Value' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2023-06-05 13:44:45,514 INFO Added plugin 'org.apache.kafka.connect.transforms.TimestampRouter' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2023-06-05 13:44:45,514 INFO Added plugin 'org.apache.kafka.connect.transforms.RegexRouter' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2023-06-05 13:44:45,514 INFO Added plugin 'org.apache.kafka.connect.transforms.HoistField$Value' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2023-06-05 13:44:45,514 INFO Added plugin 'org.apache.kafka.connect.transforms.ValueToKey' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2023-06-05 13:44:45,514 INFO Added plugin 'org.apache.kafka.connect.transforms.MaskField$Key' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2023-06-05 13:44:45,514 INFO Added plugin 'org.apache.kafka.connect.transforms.DropHeaders' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2023-06-05 13:44:45,514 INFO Added plugin 'org.apache.kafka.connect.transforms.Cast$Key' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2023-06-05 13:44:45,514 INFO Added plugin 'org.apache.kafka.connect.transforms.Cast$Value' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2023-06-05 13:44:45,514 INFO Added plugin 'org.apache.kafka.connect.runtime.PredicatedTransformation' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2023-06-05 13:44:45,514 INFO Added plugin 'org.apache.kafka.connect.transforms.ExtractField$Key' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2023-06-05 13:44:45,514 INFO Added plugin 'org.apache.kafka.connect.transforms.Flatten$Value' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2023-06-05 13:44:45,514 INFO Added plugin 'org.apache.kafka.connect.transforms.InsertHeader' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2023-06-05 13:44:45,514 INFO Added plugin 'org.apache.kafka.connect.transforms.InsertField$Key' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2023-06-05 13:44:45,514 INFO Added plugin 'org.apache.kafka.connect.transforms.Flatten$Key' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2023-06-05 13:44:45,514 INFO Added plugin 'org.apache.kafka.connect.transforms.HeaderFrom$Value' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2023-06-05 13:44:45,514 INFO Added plugin 'org.apache.kafka.connect.transforms.SetSchemaMetadata$Key' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2023-06-05 13:44:45,514 INFO Added plugin 'org.apache.kafka.connect.transforms.ExtractField$Value' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2023-06-05 13:44:45,514 INFO Added plugin 'org.apache.kafka.connect.transforms.TimestampConverter$Value' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2023-06-05 13:44:45,514 INFO Added plugin 'org.apache.kafka.connect.transforms.HoistField$Key' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2023-06-05 13:44:45,515 INFO Added plugin 'org.apache.kafka.connect.transforms.predicates.HasHeaderKey' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2023-06-05 13:44:45,515 INFO Added plugin 'org.apache.kafka.connect.transforms.predicates.RecordIsTombstone' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2023-06-05 13:44:45,515 INFO Added plugin 'org.apache.kafka.connect.transforms.predicates.TopicNameMatches' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2023-06-05 13:44:45,515 INFO Added plugin 'io.strimzi.kafka.EnvVarConfigProvider' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2023-06-05 13:44:45,515 INFO Added plugin 'io.strimzi.kafka.KubernetesConfigMapConfigProvider' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2023-06-05 13:44:45,515 INFO Added plugin 'io.strimzi.kafka.KubernetesSecretConfigProvider' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2023-06-05 13:44:45,515 INFO Added plugin 'org.apache.kafka.connect.rest.basic.auth.extension.BasicAuthSecurityRestExtension' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2023-06-05 13:44:45,516 INFO Added aliases 'SnowflakeSinkConnector' and 'SnowflakeSink' to plugin 'com.snowflake.kafka.connector.SnowflakeSinkConnector' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2023-06-05 13:44:45,516 INFO Added aliases 'JdbcSinkConnector' and 'JdbcSink' to plugin 'io.confluent.connect.jdbc.JdbcSinkConnector' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2023-06-05 13:44:45,516 INFO Added aliases 'JdbcSourceConnector' and 'JdbcSource' to plugin 'io.confluent.connect.jdbc.JdbcSourceConnector' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2023-06-05 13:44:45,516 INFO Added aliases 'SqlServerConnector' and 'SqlServer' to plugin 'io.debezium.connector.sqlserver.SqlServerConnector' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2023-06-05 13:44:45,516 INFO Added aliases 'MirrorCheckpointConnector' and 'MirrorCheckpoint' to plugin 'org.apache.kafka.connect.mirror.MirrorCheckpointConnector' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2023-06-05 13:44:45,517 INFO Added aliases 'MirrorHeartbeatConnector' and 'MirrorHeartbeat' to plugin 'org.apache.kafka.connect.mirror.MirrorHeartbeatConnector' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2023-06-05 13:44:45,517 INFO Added aliases 'MirrorSourceConnector' and 'MirrorSource' to plugin 'org.apache.kafka.connect.mirror.MirrorSourceConnector' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2023-06-05 13:44:45,517 INFO Added aliases 'MockSinkConnector' and 'MockSink' to plugin 'org.apache.kafka.connect.tools.MockSinkConnector' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2023-06-05 13:44:45,517 INFO Added aliases 'MockSourceConnector' and 'MockSource' to plugin 'org.apache.kafka.connect.tools.MockSourceConnector' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2023-06-05 13:44:45,517 INFO Added aliases 'SchemaSourceConnector' and 'SchemaSource' to plugin 'org.apache.kafka.connect.tools.SchemaSourceConnector' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2023-06-05 13:44:45,517 INFO Added aliases 'VerifiableSinkConnector' and 'VerifiableSink' to plugin 'org.apache.kafka.connect.tools.VerifiableSinkConnector' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2023-06-05 13:44:45,517 INFO Added aliases 'VerifiableSourceConnector' and 'VerifiableSource' to plugin 'org.apache.kafka.connect.tools.VerifiableSourceConnector' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2023-06-05 13:44:45,517 INFO Added aliases 'SnowflakeJsonConverter' and 'SnowflakeJson' to plugin 'com.snowflake.kafka.connector.records.SnowflakeJsonConverter' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2023-06-05 13:44:45,517 INFO Added aliases 'AvroConverter' and 'Avro' to plugin 'io.confluent.connect.avro.AvroConverter' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2023-06-05 13:44:45,518 INFO Added aliases 'BinaryDataConverter' and 'BinaryData' to plugin 'io.debezium.converters.BinaryDataConverter' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2023-06-05 13:44:45,518 INFO Added aliases 'CloudEventsConverter' and 'CloudEvents' to plugin 'io.debezium.converters.CloudEventsConverter' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2023-06-05 13:44:45,518 INFO Added aliases 'DoubleConverter' and 'Double' to plugin 'org.apache.kafka.connect.converters.DoubleConverter' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2023-06-05 13:44:45,518 INFO Added aliases 'FloatConverter' and 'Float' to plugin 'org.apache.kafka.connect.converters.FloatConverter' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2023-06-05 13:44:45,519 INFO Added aliases 'IntegerConverter' and 'Integer' to plugin 'org.apache.kafka.connect.converters.IntegerConverter' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2023-06-05 13:44:45,519 INFO Added aliases 'LongConverter' and 'Long' to plugin 'org.apache.kafka.connect.converters.LongConverter' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2023-06-05 13:44:45,519 INFO Added aliases 'ShortConverter' and 'Short' to plugin 'org.apache.kafka.connect.converters.ShortConverter' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2023-06-05 13:44:45,519 INFO Added aliases 'JsonConverter' and 'Json' to plugin 'org.apache.kafka.connect.json.JsonConverter' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2023-06-05 13:44:45,519 INFO Added aliases 'StringConverter' and 'String' to plugin 'org.apache.kafka.connect.storage.StringConverter' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2023-06-05 13:44:45,519 INFO Added aliases 'BinaryDataConverter' and 'BinaryData' to plugin 'io.debezium.converters.BinaryDataConverter' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2023-06-05 13:44:45,519 INFO Added aliases 'DoubleConverter' and 'Double' to plugin 'org.apache.kafka.connect.converters.DoubleConverter' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2023-06-05 13:44:45,519 INFO Added aliases 'FloatConverter' and 'Float' to plugin 'org.apache.kafka.connect.converters.FloatConverter' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2023-06-05 13:44:45,519 INFO Added aliases 'IntegerConverter' and 'Integer' to plugin 'org.apache.kafka.connect.converters.IntegerConverter' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2023-06-05 13:44:45,520 INFO Added aliases 'LongConverter' and 'Long' to plugin 'org.apache.kafka.connect.converters.LongConverter' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2023-06-05 13:44:45,520 INFO Added aliases 'ShortConverter' and 'Short' to plugin 'org.apache.kafka.connect.converters.ShortConverter' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2023-06-05 13:44:45,520 INFO Added aliases 'JsonConverter' and 'Json' to plugin 'org.apache.kafka.connect.json.JsonConverter' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2023-06-05 13:44:45,520 INFO Added aliases 'SimpleHeaderConverter' and 'Simple' to plugin 'org.apache.kafka.connect.storage.SimpleHeaderConverter' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2023-06-05 13:44:45,520 INFO Added aliases 'StringConverter' and 'String' to plugin 'org.apache.kafka.connect.storage.StringConverter' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2023-06-05 13:44:45,520 INFO Added alias 'ByLogicalTableRouter' to plugin 'io.debezium.transforms.ByLogicalTableRouter' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2023-06-05 13:44:45,521 INFO Added alias 'ExtractChangedRecordState' to plugin 'io.debezium.transforms.ExtractChangedRecordState' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2023-06-05 13:44:45,521 INFO Added alias 'ExtractNewRecordState' to plugin 'io.debezium.transforms.ExtractNewRecordState' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2023-06-05 13:44:45,521 INFO Added alias 'HeaderToValue' to plugin 'io.debezium.transforms.HeaderToValue' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2023-06-05 13:44:45,521 INFO Added alias 'EventRouter' to plugin 'io.debezium.transforms.outbox.EventRouter' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2023-06-05 13:44:45,521 INFO Added alias 'ComputePartition' to plugin 'io.debezium.transforms.partitions.ComputePartition' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2023-06-05 13:44:45,521 INFO Added alias 'PartitionRouting' to plugin 'io.debezium.transforms.partitions.PartitionRouting' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2023-06-05 13:44:45,522 INFO Added alias 'ActivateTracingSpan' to plugin 'io.debezium.transforms.tracing.ActivateTracingSpan' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2023-06-05 13:44:45,522 INFO Added aliases 'PredicatedTransformation' and 'Predicated' to plugin 'org.apache.kafka.connect.runtime.PredicatedTransformation' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2023-06-05 13:44:45,522 INFO Added alias 'DropHeaders' to plugin 'org.apache.kafka.connect.transforms.DropHeaders' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2023-06-05 13:44:45,522 INFO Added alias 'Filter' to plugin 'org.apache.kafka.connect.transforms.Filter' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2023-06-05 13:44:45,523 INFO Added alias 'InsertHeader' to plugin 'org.apache.kafka.connect.transforms.InsertHeader' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2023-06-05 13:44:45,523 INFO Added alias 'RegexRouter' to plugin 'org.apache.kafka.connect.transforms.RegexRouter' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2023-06-05 13:44:45,524 INFO Added alias 'TimestampRouter' to plugin 'org.apache.kafka.connect.transforms.TimestampRouter' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2023-06-05 13:44:45,524 INFO Added alias 'ValueToKey' to plugin 'org.apache.kafka.connect.transforms.ValueToKey' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2023-06-05 13:44:45,524 INFO Added alias 'HasHeaderKey' to plugin 'org.apache.kafka.connect.transforms.predicates.HasHeaderKey' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2023-06-05 13:44:45,524 INFO Added alias 'RecordIsTombstone' to plugin 'org.apache.kafka.connect.transforms.predicates.RecordIsTombstone' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2023-06-05 13:44:45,524 INFO Added alias 'TopicNameMatches' to plugin 'org.apache.kafka.connect.transforms.predicates.TopicNameMatches' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2023-06-05 13:44:45,524 INFO Added alias 'BasicAuthSecurityRestExtension' to plugin 'org.apache.kafka.connect.rest.basic.auth.extension.BasicAuthSecurityRestExtension' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2023-06-05 13:44:45,525 INFO Added aliases 'AllConnectorClientConfigOverridePolicy' and 'All' to plugin 'org.apache.kafka.connect.connector.policy.AllConnectorClientConfigOverridePolicy' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2023-06-05 13:44:45,525 INFO Added aliases 'NoneConnectorClientConfigOverridePolicy' and 'None' to plugin 'org.apache.kafka.connect.connector.policy.NoneConnectorClientConfigOverridePolicy' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2023-06-05 13:44:45,525 INFO Added aliases 'PrincipalConnectorClientConfigOverridePolicy' and 'Principal' to plugin 'org.apache.kafka.connect.connector.policy.PrincipalConnectorClientConfigOverridePolicy' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2023-06-05 13:44:45,824 INFO DistributedConfig values: 
	access.control.allow.methods = 
	access.control.allow.origin = 
	admin.listeners = null
	auto.include.jmx.reporter = true
	bootstrap.servers = [evh-pars-dev-centralus.servicebus.windows.net:9093]
	client.dns.lookup = use_all_dns_ips
	client.id = 
	config.providers = []
	config.storage.replication.factor = 3
	config.storage.topic = strimzi-connect-cluster-configs
	connect.protocol = sessioned
	connections.max.idle.ms = 540000
	connector.client.config.override.policy = All
	exactly.once.source.support = disabled
	group.id = connect-cluster
	header.converter = class org.apache.kafka.connect.storage.SimpleHeaderConverter
	heartbeat.interval.ms = 3000
	inter.worker.key.generation.algorithm = HmacSHA256
	inter.worker.key.size = null
	inter.worker.key.ttl.ms = 3600000
	inter.worker.signature.algorithm = HmacSHA256
	inter.worker.verification.algorithms = [HmacSHA256]
	key.converter = class org.apache.kafka.connect.json.JsonConverter
	listeners = [http://:8083]
	metadata.max.age.ms = 300000
	metric.reporters = []
	metrics.num.samples = 2
	metrics.recording.level = INFO
	metrics.sample.window.ms = 30000
	offset.flush.interval.ms = 60000
	offset.flush.timeout.ms = 5000
	offset.storage.partitions = 25
	offset.storage.replication.factor = 3
	offset.storage.topic = strimzi-connect-cluster-offsets
	plugin.path = [/opt/kafka/plugins]
	rebalance.timeout.ms = 60000
	receive.buffer.bytes = 32768
	reconnect.backoff.max.ms = 1000
	reconnect.backoff.ms = 50
	request.timeout.ms = 60000
	response.http.headers.config = 
	rest.advertised.host.name = 10.244.0.33
	rest.advertised.listener = null
	rest.advertised.port = 8083
	rest.extension.classes = []
	retry.backoff.ms = 100
	sasl.client.callback.handler.class = null
	sasl.jaas.config = [hidden]
	sasl.kerberos.kinit.cmd = /usr/bin/kinit
	sasl.kerberos.min.time.before.relogin = 60000
	sasl.kerberos.service.name = null
	sasl.kerberos.ticket.renew.jitter = 0.05
	sasl.kerberos.ticket.renew.window.factor = 0.8
	sasl.login.callback.handler.class = null
	sasl.login.class = null
	sasl.login.connect.timeout.ms = null
	sasl.login.read.timeout.ms = null
	sasl.login.refresh.buffer.seconds = 300
	sasl.login.refresh.min.period.seconds = 60
	sasl.login.refresh.window.factor = 0.8
	sasl.login.refresh.window.jitter = 0.05
	sasl.login.retry.backoff.max.ms = 10000
	sasl.login.retry.backoff.ms = 100
	sasl.mechanism = PLAIN
	sasl.oauthbearer.clock.skew.seconds = 30
	sasl.oauthbearer.expected.audience = null
	sasl.oauthbearer.expected.issuer = null
	sasl.oauthbearer.jwks.endpoint.refresh.ms = 3600000
	sasl.oauthbearer.jwks.endpoint.retry.backoff.max.ms = 10000
	sasl.oauthbearer.jwks.endpoint.retry.backoff.ms = 100
	sasl.oauthbearer.jwks.endpoint.url = null
	sasl.oauthbearer.scope.claim.name = scope
	sasl.oauthbearer.sub.claim.name = sub
	sasl.oauthbearer.token.endpoint.url = null
	scheduled.rebalance.max.delay.ms = 300000
	security.protocol = SASL_SSL
	send.buffer.bytes = 131072
	session.timeout.ms = 10000
	socket.connection.setup.timeout.max.ms = 30000
	socket.connection.setup.timeout.ms = 10000
	ssl.cipher.suites = null
	ssl.client.auth = none
	ssl.enabled.protocols = [TLSv1.2, TLSv1.3]
	ssl.endpoint.identification.algorithm = https
	ssl.engine.factory.class = null
	ssl.key.password = null
	ssl.keymanager.algorithm = SunX509
	ssl.keystore.certificate.chain = null
	ssl.keystore.key = null
	ssl.keystore.location = null
	ssl.keystore.password = null
	ssl.keystore.type = JKS
	ssl.protocol = TLSv1.3
	ssl.provider = null
	ssl.secure.random.implementation = null
	ssl.trustmanager.algorithm = PKIX
	ssl.truststore.certificates = null
	ssl.truststore.location = null
	ssl.truststore.password = null
	ssl.truststore.type = JKS
	status.storage.partitions = 5
	status.storage.replication.factor = 3
	status.storage.topic = strimzi-connect-cluster-status
	task.shutdown.graceful.timeout.ms = 5000
	topic.creation.enable = true
	topic.tracking.allow.reset = true
	topic.tracking.enable = true
	value.converter = class org.apache.kafka.connect.json.JsonConverter
	worker.sync.timeout.ms = 3000
	worker.unsync.backoff.ms = 300000
 (org.apache.kafka.connect.runtime.distributed.DistributedConfig) [main]
2023-06-05 13:44:45,826 INFO Creating Kafka admin client (org.apache.kafka.connect.runtime.WorkerConfig) [main]
2023-06-05 13:44:45,829 INFO AdminClientConfig values: 
	auto.include.jmx.reporter = true
	bootstrap.servers = [evh-pars-dev-centralus.servicebus.windows.net:9093]
	client.dns.lookup = use_all_dns_ips
	client.id = 
	connections.max.idle.ms = 300000
	default.api.timeout.ms = 60000
	metadata.max.age.ms = 300000
	metric.reporters = []
	metrics.num.samples = 2
	metrics.recording.level = INFO
	metrics.sample.window.ms = 30000
	receive.buffer.bytes = 65536
	reconnect.backoff.max.ms = 1000
	reconnect.backoff.ms = 50
	request.timeout.ms = 60000
	retries = 2147483647
	retry.backoff.ms = 100
	sasl.client.callback.handler.class = null
	sasl.jaas.config = [hidden]
	sasl.kerberos.kinit.cmd = /usr/bin/kinit
	sasl.kerberos.min.time.before.relogin = 60000
	sasl.kerberos.service.name = null
	sasl.kerberos.ticket.renew.jitter = 0.05
	sasl.kerberos.ticket.renew.window.factor = 0.8
	sasl.login.callback.handler.class = null
	sasl.login.class = null
	sasl.login.connect.timeout.ms = null
	sasl.login.read.timeout.ms = null
	sasl.login.refresh.buffer.seconds = 300
	sasl.login.refresh.min.period.seconds = 60
	sasl.login.refresh.window.factor = 0.8
	sasl.login.refresh.window.jitter = 0.05
	sasl.login.retry.backoff.max.ms = 10000
	sasl.login.retry.backoff.ms = 100
	sasl.mechanism = PLAIN
	sasl.oauthbearer.clock.skew.seconds = 30
	sasl.oauthbearer.expected.audience = null
	sasl.oauthbearer.expected.issuer = null
	sasl.oauthbearer.jwks.endpoint.refresh.ms = 3600000
	sasl.oauthbearer.jwks.endpoint.retry.backoff.max.ms = 10000
	sasl.oauthbearer.jwks.endpoint.retry.backoff.ms = 100
	sasl.oauthbearer.jwks.endpoint.url = null
	sasl.oauthbearer.scope.claim.name = scope
	sasl.oauthbearer.sub.claim.name = sub
	sasl.oauthbearer.token.endpoint.url = null
	security.protocol = SASL_SSL
	security.providers = null
	send.buffer.bytes = 131072
	socket.connection.setup.timeout.max.ms = 30000
	socket.connection.setup.timeout.ms = 10000
	ssl.cipher.suites = null
	ssl.enabled.protocols = [TLSv1.2, TLSv1.3]
	ssl.endpoint.identification.algorithm = https
	ssl.engine.factory.class = null
	ssl.key.password = null
	ssl.keymanager.algorithm = SunX509
	ssl.keystore.certificate.chain = null
	ssl.keystore.key = null
	ssl.keystore.location = null
	ssl.keystore.password = null
	ssl.keystore.type = JKS
	ssl.protocol = TLSv1.3
	ssl.provider = null
	ssl.secure.random.implementation = null
	ssl.trustmanager.algorithm = PKIX
	ssl.truststore.certificates = null
	ssl.truststore.location = null
	ssl.truststore.password = null
	ssl.truststore.type = JKS
 (org.apache.kafka.clients.admin.AdminClientConfig) [main]
2023-06-05 13:44:46,219 INFO Successfully logged in. (org.apache.kafka.common.security.authenticator.AbstractLogin) [main]
2023-06-05 13:44:46,715 WARN These configurations '[producer.sasl.jaas.config, offset.flush.internal.ms, group.id, rest.advertised.port, plugin.path, consumer.sasl.mechanism, admin.security.protocol, kafka.heap.opts, offset.storage.topic, consumer.security.protocol, admin.sasl.mechanism, value.converter, key.converter, consumer.client.rack, consumer.sasl.jaas.config, config.storage.topic, producer.security.protocol, rest.advertised.host.name, status.storage.topic, group.Id, producer.sasl.mechanism, rest.port, key.converter.schemas.enable, value.converter.schemas.enable, admin.sasl.jaas.config]' were supplied but are not used yet. (org.apache.kafka.clients.admin.AdminClientConfig) [main]
2023-06-05 13:44:46,715 INFO Kafka version: 3.4.0 (org.apache.kafka.common.utils.AppInfoParser) [main]
2023-06-05 13:44:46,716 INFO Kafka commitId: 2e1947d240607d53 (org.apache.kafka.common.utils.AppInfoParser) [main]
2023-06-05 13:44:46,716 INFO Kafka startTimeMs: 1685972686715 (org.apache.kafka.common.utils.AppInfoParser) [main]
2023-06-05 13:44:55,361 INFO [AdminClient clientId=adminclient-1] Disconnecting from node -1 due to socket connection setup timeout. The timeout value is 8565 ms. (org.apache.kafka.clients.NetworkClient) [kafka-admin-client-thread | adminclient-1]
2023-06-05 13:45:15,747 INFO [AdminClient clientId=adminclient-1] Disconnecting from node -1 due to socket connection setup timeout. The timeout value is 20280 ms. (org.apache.kafka.clients.NetworkClient) [kafka-admin-client-thread | adminclient-1]

scholzj Jun 5, 2023
Maintainer

I do not see any shutdown in this log. Is it really complete?

imrohankataria Jun 5, 2023
Author

@scholzj
Thankyou for pointing that out, i just added --previous flag for logs and now able to see shutdown keyword in logs

2023-06-05 15:06:38,304 INFO [Worker clientId=connect-1, groupId=connect-cluster] (Re-)joining group (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-06-05 15:06:38,319 INFO [Worker clientId=connect-1, groupId=connect-cluster] Successfully joined group with generation Generation{generationId=28877, memberId='evh-pars-dev-centralus.servicebus.windows.net:c:connect-cluster:I:connect-1-805b149901cf40948dfdab5f8f568065', protocol='sessioned'} (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-06-05 15:06:38,335 INFO [Worker clientId=connect-1, groupId=connect-cluster] Successfully synced group in generation Generation{generationId=28877, memberId='evh-pars-dev-centralus.servicebus.windows.net:c:connect-cluster:I:connect-1-805b149901cf40948dfdab5f8f568065', protocol='sessioned'} (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-06-05 15:06:38,335 INFO [Worker clientId=connect-1, groupId=connect-cluster] Joined group at generation 28877 with protocol version 2 and got assignment: Assignment{error=1, leader='evh-pars-dev-centralus.servicebus.windows.net:c:connect-cluster:I:connect-1-f2ebe64c8c3a49b09a7db33b57713f81', leaderUrl='http://10.244.15.4:8083/', offset=669, connectorIds=[], taskIds=[], revokedConnectorIds=[], revokedTaskIds=[], delay=0} with rebalance delay: 0 (org.apache.kafka.connect.runtime.distributed.DistributedHerder) [DistributedHerder-connect-1-1]
2023-06-05 15:06:38,335 INFO [Worker clientId=connect-1, groupId=connect-cluster] Rebalance started (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-06-05 15:06:38,335 INFO [Worker clientId=connect-1, groupId=connect-cluster] (Re-)joining group (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-06-05 15:06:38,351 INFO [Worker clientId=connect-1, groupId=connect-cluster] Successfully joined group with generation Generation{generationId=28878, memberId='evh-pars-dev-centralus.servicebus.windows.net:c:connect-cluster:I:connect-1-805b149901cf40948dfdab5f8f568065', protocol='sessioned'} (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-06-05 15:06:38,366 INFO [Worker clientId=connect-1, groupId=connect-cluster] Successfully synced group in generation Generation{generationId=28878, memberId='evh-pars-dev-centralus.servicebus.windows.net:c:connect-cluster:I:connect-1-805b149901cf40948dfdab5f8f568065', protocol='sessioned'} (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-06-05 15:06:38,366 INFO [Worker clientId=connect-1, groupId=connect-cluster] Joined group at generation 28878 with protocol version 2 and got assignment: Assignment{error=1, leader='evh-pars-dev-centralus.servicebus.windows.net:c:connect-cluster:I:connect-1-f2ebe64c8c3a49b09a7db33b57713f81', leaderUrl='http://10.244.15.4:8083/', offset=669, connectorIds=[], taskIds=[], revokedConnectorIds=[], revokedTaskIds=[], delay=0} with rebalance delay: 0 (org.apache.kafka.connect.runtime.distributed.DistributedHerder) [DistributedHerder-connect-1-1]
2023-06-05 15:06:38,366 INFO [Worker clientId=connect-1, groupId=connect-cluster] Rebalance started (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-06-05 15:06:38,366 INFO [Worker clientId=connect-1, groupId=connect-cluster] (Re-)joining group (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-06-05 15:06:38,489 INFO [Worker clientId=connect-1, groupId=connect-cluster] Successfully joined group with generation Generation{generationId=28879, memberId='evh-pars-dev-centralus.servicebus.windows.net:c:connect-cluster:I:connect-1-805b149901cf40948dfdab5f8f568065', protocol='sessioned'} (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-06-05 15:06:38,498 INFO [Worker clientId=connect-1, groupId=connect-cluster] Successfully synced group in generation Generation{generationId=28879, memberId='evh-pars-dev-centralus.servicebus.windows.net:c:connect-cluster:I:connect-1-805b149901cf40948dfdab5f8f568065', protocol='sessioned'} (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-06-05 15:06:38,498 INFO [Worker clientId=connect-1, groupId=connect-cluster] Joined group at generation 28879 with protocol version 2 and got assignment: Assignment{error=1, leader='evh-pars-dev-centralus.servicebus.windows.net:c:connect-cluster:I:connect-1-f2ebe64c8c3a49b09a7db33b57713f81', leaderUrl='http://10.244.15.4:8083/', offset=669, connectorIds=[], taskIds=[], revokedConnectorIds=[], revokedTaskIds=[], delay=0} with rebalance delay: 0 (org.apache.kafka.connect.runtime.distributed.DistributedHerder) [DistributedHerder-connect-1-1]
2023-06-05 15:06:38,498 INFO [Worker clientId=connect-1, groupId=connect-cluster] Rebalance started (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-06-05 15:06:38,498 INFO [Worker clientId=connect-1, groupId=connect-cluster] (Re-)joining group (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-06-05 15:06:38,507 INFO [Worker clientId=connect-1, groupId=connect-cluster] Successfully joined group with generation Generation{generationId=28879, memberId='evh-pars-dev-centralus.servicebus.windows.net:c:connect-cluster:I:connect-1-805b149901cf40948dfdab5f8f568065', protocol='sessioned'} (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-06-05 15:06:38,523 INFO [Worker clientId=connect-1, groupId=connect-cluster] SyncGroup failed: The group began another rebalance. Need to re-join the group. Sent generation was Generation{generationId=28879, memberId='evh-pars-dev-centralus.servicebus.windows.net:c:connect-cluster:I:connect-1-805b149901cf40948dfdab5f8f568065', protocol='sessioned'} (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-06-05 15:06:38,523 INFO [Worker clientId=connect-1, groupId=connect-cluster] Request joining group due to: rebalance failed due to 'The group is rebalancing, so a rejoin is needed.' (RebalanceInProgressException) (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-06-05 15:06:38,523 INFO [Worker clientId=connect-1, groupId=connect-cluster] (Re-)joining group (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-06-05 15:06:38,629 INFO [Worker clientId=connect-1, groupId=connect-cluster] Successfully joined group with generation Generation{generationId=28880, memberId='evh-pars-dev-centralus.servicebus.windows.net:c:connect-cluster:I:connect-1-805b149901cf40948dfdab5f8f568065', protocol='sessioned'} (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-06-05 15:06:38,647 INFO [Worker clientId=connect-1, groupId=connect-cluster] Successfully synced group in generation Generation{generationId=28880, memberId='evh-pars-dev-centralus.servicebus.windows.net:c:connect-cluster:I:connect-1-805b149901cf40948dfdab5f8f568065', protocol='sessioned'} (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-06-05 15:06:38,647 INFO [Worker clientId=connect-1, groupId=connect-cluster] Joined group at generation 28880 with protocol version 2 and got assignment: Assignment{error=1, leader='evh-pars-dev-centralus.servicebus.windows.net:c:connect-cluster:I:connect-1-f2ebe64c8c3a49b09a7db33b57713f81', leaderUrl='http://10.244.15.4:8083/', offset=669, connectorIds=[], taskIds=[], revokedConnectorIds=[], revokedTaskIds=[], delay=0} with rebalance delay: 0 (org.apache.kafka.connect.runtime.distributed.DistributedHerder) [DistributedHerder-connect-1-1]
2023-06-05 15:06:38,647 INFO [Worker clientId=connect-1, groupId=connect-cluster] Rebalance started (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-06-05 15:06:38,647 INFO [Worker clientId=connect-1, groupId=connect-cluster] (Re-)joining group (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-06-05 15:06:38,770 INFO [Worker clientId=connect-1, groupId=connect-cluster] Successfully joined group with generation Generation{generationId=28881, memberId='evh-pars-dev-centralus.servicebus.windows.net:c:connect-cluster:I:connect-1-805b149901cf40948dfdab5f8f568065', protocol='sessioned'} (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-06-05 15:06:38,788 INFO [Worker clientId=connect-1, groupId=connect-cluster] Successfully synced group in generation Generation{generationId=28881, memberId='evh-pars-dev-centralus.servicebus.windows.net:c:connect-cluster:I:connect-1-805b149901cf40948dfdab5f8f568065', protocol='sessioned'} (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-06-05 15:06:38,788 INFO [Worker clientId=connect-1, groupId=connect-cluster] Joined group at generation 28881 with protocol version 2 and got assignment: Assignment{error=1, leader='evh-pars-dev-centralus.servicebus.windows.net:c:connect-cluster:I:connect-1-f2ebe64c8c3a49b09a7db33b57713f81', leaderUrl='http://10.244.15.4:8083/', offset=669, connectorIds=[], taskIds=[], revokedConnectorIds=[], revokedTaskIds=[], delay=0} with rebalance delay: 0 (org.apache.kafka.connect.runtime.distributed.DistributedHerder) [DistributedHerder-connect-1-1]
2023-06-05 15:06:38,788 INFO [Worker clientId=connect-1, groupId=connect-cluster] Rebalance started (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-06-05 15:06:38,788 INFO [Worker clientId=connect-1, groupId=connect-cluster] (Re-)joining group (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-06-05 15:06:38,804 INFO [Worker clientId=connect-1, groupId=connect-cluster] Successfully joined group with generation Generation{generationId=28882, memberId='evh-pars-dev-centralus.servicebus.windows.net:c:connect-cluster:I:connect-1-805b149901cf40948dfdab5f8f568065', protocol='sessioned'} (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-06-05 15:06:38,819 INFO [Worker clientId=connect-1, groupId=connect-cluster] Successfully synced group in generation Generation{generationId=28882, memberId='evh-pars-dev-centralus.servicebus.windows.net:c:connect-cluster:I:connect-1-805b149901cf40948dfdab5f8f568065', protocol='sessioned'} (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-06-05 15:06:38,819 INFO [Worker clientId=connect-1, groupId=connect-cluster] Joined group at generation 28882 with protocol version 2 and got assignment: Assignment{error=1, leader='evh-pars-dev-centralus.servicebus.windows.net:c:connect-cluster:I:connect-1-f2ebe64c8c3a49b09a7db33b57713f81', leaderUrl='http://10.244.15.4:8083/', offset=669, connectorIds=[], taskIds=[], revokedConnectorIds=[], revokedTaskIds=[], delay=0} with rebalance delay: 0 (org.apache.kafka.connect.runtime.distributed.DistributedHerder) [DistributedHerder-connect-1-1]
2023-06-05 15:06:38,819 INFO [Worker clientId=connect-1, groupId=connect-cluster] Rebalance started (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-06-05 15:06:38,819 INFO [Worker clientId=connect-1, groupId=connect-cluster] (Re-)joining group (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-06-05 15:06:38,835 INFO [Worker clientId=connect-1, groupId=connect-cluster] Successfully joined group with generation Generation{generationId=28883, memberId='evh-pars-dev-centralus.servicebus.windows.net:c:connect-cluster:I:connect-1-805b149901cf40948dfdab5f8f568065', protocol='sessioned'} (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-06-05 15:06:38,850 INFO [Worker clientId=connect-1, groupId=connect-cluster] Successfully synced group in generation Generation{generationId=28883, memberId='evh-pars-dev-centralus.servicebus.windows.net:c:connect-cluster:I:connect-1-805b149901cf40948dfdab5f8f568065', protocol='sessioned'} (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-06-05 15:06:38,850 INFO [Worker clientId=connect-1, groupId=connect-cluster] Joined group at generation 28883 with protocol version 2 and got assignment: Assignment{error=1, leader='evh-pars-dev-centralus.servicebus.windows.net:c:connect-cluster:I:connect-1-f2ebe64c8c3a49b09a7db33b57713f81', leaderUrl='http://10.244.15.4:8083/', offset=669, connectorIds=[], taskIds=[], revokedConnectorIds=[], revokedTaskIds=[], delay=0} with rebalance delay: 0 (org.apache.kafka.connect.runtime.distributed.DistributedHerder) [DistributedHerder-connect-1-1]
2023-06-05 15:06:38,850 INFO [Worker clientId=connect-1, groupId=connect-cluster] Rebalance started (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-06-05 15:06:38,850 INFO [Worker clientId=connect-1, groupId=connect-cluster] (Re-)joining group (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-06-05 15:06:38,958 INFO [Worker clientId=connect-1, groupId=connect-cluster] Successfully joined group with generation Generation{generationId=28884, memberId='evh-pars-dev-centralus.servicebus.windows.net:c:connect-cluster:I:connect-1-805b149901cf40948dfdab5f8f568065', protocol='sessioned'} (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-06-05 15:06:38,976 INFO [Worker clientId=connect-1, groupId=connect-cluster] Successfully synced group in generation Generation{generationId=28884, memberId='evh-pars-dev-centralus.servicebus.windows.net:c:connect-cluster:I:connect-1-805b149901cf40948dfdab5f8f568065', protocol='sessioned'} (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-06-05 15:06:38,976 INFO [Worker clientId=connect-1, groupId=connect-cluster] Joined group at generation 28884 with protocol version 2 and got assignment: Assignment{error=1, leader='evh-pars-dev-centralus.servicebus.windows.net:c:connect-cluster:I:connect-1-f2ebe64c8c3a49b09a7db33b57713f81', leaderUrl='http://10.244.15.4:8083/', offset=669, connectorIds=[], taskIds=[], revokedConnectorIds=[], revokedTaskIds=[], delay=0} with rebalance delay: 0 (org.apache.kafka.connect.runtime.distributed.DistributedHerder) [DistributedHerder-connect-1-1]
2023-06-05 15:06:38,976 INFO [Worker clientId=connect-1, groupId=connect-cluster] Rebalance started (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-06-05 15:06:38,976 INFO [Worker clientId=connect-1, groupId=connect-cluster] (Re-)joining group (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-06-05 15:06:39,098 INFO [Worker clientId=connect-1, groupId=connect-cluster] Successfully joined group with generation Generation{generationId=28885, memberId='evh-pars-dev-centralus.servicebus.windows.net:c:connect-cluster:I:connect-1-805b149901cf40948dfdab5f8f568065', protocol='sessioned'} (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-06-05 15:06:39,117 INFO [Worker clientId=connect-1, groupId=connect-cluster] Successfully synced group in generation Generation{generationId=28885, memberId='evh-pars-dev-centralus.servicebus.windows.net:c:connect-cluster:I:connect-1-805b149901cf40948dfdab5f8f568065', protocol='sessioned'} (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-06-05 15:06:39,117 INFO [Worker clientId=connect-1, groupId=connect-cluster] Joined group at generation 28885 with protocol version 2 and got assignment: Assignment{error=1, leader='evh-pars-dev-centralus.servicebus.windows.net:c:connect-cluster:I:connect-1-f2ebe64c8c3a49b09a7db33b57713f81', leaderUrl='http://10.244.15.4:8083/', offset=669, connectorIds=[], taskIds=[], revokedConnectorIds=[], revokedTaskIds=[], delay=0} with rebalance delay: 0 (org.apache.kafka.connect.runtime.distributed.DistributedHerder) [DistributedHerder-connect-1-1]
2023-06-05 15:06:39,117 INFO [Worker clientId=connect-1, groupId=connect-cluster] Rebalance started (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-06-05 15:06:39,117 INFO [Worker clientId=connect-1, groupId=connect-cluster] (Re-)joining group (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-06-05 15:06:39,239 INFO [Worker clientId=connect-1, groupId=connect-cluster] Successfully joined group with generation Generation{generationId=28886, memberId='evh-pars-dev-centralus.servicebus.windows.net:c:connect-cluster:I:connect-1-805b149901cf40948dfdab5f8f568065', protocol='sessioned'} (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-06-05 15:06:39,257 INFO [Worker clientId=connect-1, groupId=connect-cluster] Successfully synced group in generation Generation{generationId=28886, memberId='evh-pars-dev-centralus.servicebus.windows.net:c:connect-cluster:I:connect-1-805b149901cf40948dfdab5f8f568065', protocol='sessioned'} (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-06-05 15:06:39,257 INFO [Worker clientId=connect-1, groupId=connect-cluster] Joined group at generation 28886 with protocol version 2 and got assignment: Assignment{error=1, leader='evh-pars-dev-centralus.servicebus.windows.net:c:connect-cluster:I:connect-1-f2ebe64c8c3a49b09a7db33b57713f81', leaderUrl='http://10.244.15.4:8083/', offset=669, connectorIds=[], taskIds=[], revokedConnectorIds=[], revokedTaskIds=[], delay=0} with rebalance delay: 0 (org.apache.kafka.connect.runtime.distributed.DistributedHerder) [DistributedHerder-connect-1-1]
2023-06-05 15:06:39,257 INFO [Worker clientId=connect-1, groupId=connect-cluster] Rebalance started (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-06-05 15:06:39,257 INFO [Worker clientId=connect-1, groupId=connect-cluster] (Re-)joining group (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-06-05 15:06:39,273 INFO [Worker clientId=connect-1, groupId=connect-cluster] Successfully joined group with generation Generation{generationId=28887, memberId='evh-pars-dev-centralus.servicebus.windows.net:c:connect-cluster:I:connect-1-805b149901cf40948dfdab5f8f568065', protocol='sessioned'} (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-06-05 15:06:39,288 INFO [Worker clientId=connect-1, groupId=connect-cluster] Successfully synced group in generation Generation{generationId=28887, memberId='evh-pars-dev-centralus.servicebus.windows.net:c:connect-cluster:I:connect-1-805b149901cf40948dfdab5f8f568065', protocol='sessioned'} (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-06-05 15:06:39,288 INFO [Worker clientId=connect-1, groupId=connect-cluster] Joined group at generation 28887 with protocol version 2 and got assignment: Assignment{error=1, leader='evh-pars-dev-centralus.servicebus.windows.net:c:connect-cluster:I:connect-1-f2ebe64c8c3a49b09a7db33b57713f81', leaderUrl='http://10.244.15.4:8083/', offset=669, connectorIds=[], taskIds=[], revokedConnectorIds=[], revokedTaskIds=[], delay=0} with rebalance delay: 0 (org.apache.kafka.connect.runtime.distributed.DistributedHerder) [DistributedHerder-connect-1-1]
2023-06-05 15:06:39,288 INFO [Worker clientId=connect-1, groupId=connect-cluster] Rebalance started (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-06-05 15:06:39,288 INFO [Worker clientId=connect-1, groupId=connect-cluster] (Re-)joining group (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-06-05 15:06:39,304 INFO [Worker clientId=connect-1, groupId=connect-cluster] Successfully joined group with generation Generation{generationId=28888, memberId='evh-pars-dev-centralus.servicebus.windows.net:c:connect-cluster:I:connect-1-805b149901cf40948dfdab5f8f568065', protocol='sessioned'} (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-06-05 15:06:39,319 INFO [Worker clientId=connect-1, groupId=connect-cluster] Successfully synced group in generation Generation{generationId=28888, memberId='evh-pars-dev-centralus.servicebus.windows.net:c:connect-cluster:I:connect-1-805b149901cf40948dfdab5f8f568065', protocol='sessioned'} (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-06-05 15:06:39,319 INFO [Worker clientId=connect-1, groupId=connect-cluster] Joined group at generation 28888 with protocol version 2 and got assignment: Assignment{error=1, leader='evh-pars-dev-centralus.servicebus.windows.net:c:connect-cluster:I:connect-1-f2ebe64c8c3a49b09a7db33b57713f81', leaderUrl='http://10.244.15.4:8083/', offset=669, connectorIds=[], taskIds=[], revokedConnectorIds=[], revokedTaskIds=[], delay=0} with rebalance delay: 0 (org.apache.kafka.connect.runtime.distributed.DistributedHerder) [DistributedHerder-connect-1-1]
2023-06-05 15:06:39,319 INFO [Worker clientId=connect-1, groupId=connect-cluster] Rebalance started (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-06-05 15:06:39,319 INFO [Worker clientId=connect-1, groupId=connect-cluster] (Re-)joining group (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-06-05 15:06:39,411 INFO [Worker clientId=connect-1, groupId=connect-cluster] Successfully joined group with generation Generation{generationId=28888, memberId='evh-pars-dev-centralus.servicebus.windows.net:c:connect-cluster:I:connect-1-805b149901cf40948dfdab5f8f568065', protocol='sessioned'} (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-06-05 15:06:39,429 INFO [Worker clientId=connect-1, groupId=connect-cluster] SyncGroup failed: The group began another rebalance. Need to re-join the group. Sent generation was Generation{generationId=28888, memberId='evh-pars-dev-centralus.servicebus.windows.net:c:connect-cluster:I:connect-1-805b149901cf40948dfdab5f8f568065', protocol='sessioned'} (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-06-05 15:06:39,429 INFO [Worker clientId=connect-1, groupId=connect-cluster] Request joining group due to: rebalance failed due to 'The group is rebalancing, so a rejoin is needed.' (RebalanceInProgressException) (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-06-05 15:06:39,429 INFO [Worker clientId=connect-1, groupId=connect-cluster] (Re-)joining group (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-06-05 15:06:39,445 INFO [Worker clientId=connect-1, groupId=connect-cluster] Successfully joined group with generation Generation{generationId=28889, memberId='evh-pars-dev-centralus.servicebus.windows.net:c:connect-cluster:I:connect-1-805b149901cf40948dfdab5f8f568065', protocol='sessioned'} (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-06-05 15:06:39,460 INFO [Worker clientId=connect-1, groupId=connect-cluster] Successfully synced group in generation Generation{generationId=28889, memberId='evh-pars-dev-centralus.servicebus.windows.net:c:connect-cluster:I:connect-1-805b149901cf40948dfdab5f8f568065', protocol='sessioned'} (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-06-05 15:06:39,460 INFO [Worker clientId=connect-1, groupId=connect-cluster] Joined group at generation 28889 with protocol version 2 and got assignment: Assignment{error=1, leader='evh-pars-dev-centralus.servicebus.windows.net:c:connect-cluster:I:connect-1-f2ebe64c8c3a49b09a7db33b57713f81', leaderUrl='http://10.244.15.4:8083/', offset=669, connectorIds=[], taskIds=[], revokedConnectorIds=[], revokedTaskIds=[], delay=0} with rebalance delay: 0 (org.apache.kafka.connect.runtime.distributed.DistributedHerder) [DistributedHerder-connect-1-1]
2023-06-05 15:06:39,460 INFO [Worker clientId=connect-1, groupId=connect-cluster] Rebalance started (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-06-05 15:06:39,461 INFO [Worker clientId=connect-1, groupId=connect-cluster] (Re-)joining group (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-06-05 15:06:39,476 INFO [Worker clientId=connect-1, groupId=connect-cluster] Successfully joined group with generation Generation{generationId=28890, memberId='evh-pars-dev-centralus.servicebus.windows.net:c:connect-cluster:I:connect-1-805b149901cf40948dfdab5f8f568065', protocol='sessioned'} (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-06-05 15:06:39,498 INFO [Worker clientId=connect-1, groupId=connect-cluster] Successfully synced group in generation Generation{generationId=28890, memberId='evh-pars-dev-centralus.servicebus.windows.net:c:connect-cluster:I:connect-1-805b149901cf40948dfdab5f8f568065', protocol='sessioned'} (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-06-05 15:06:39,498 INFO [Worker clientId=connect-1, groupId=connect-cluster] Joined group at generation 28890 with protocol version 2 and got assignment: Assignment{error=1, leader='evh-pars-dev-centralus.servicebus.windows.net:c:connect-cluster:I:connect-1-f2ebe64c8c3a49b09a7db33b57713f81', leaderUrl='http://10.244.15.4:8083/', offset=669, connectorIds=[], taskIds=[], revokedConnectorIds=[], revokedTaskIds=[], delay=0} with rebalance delay: 0 (org.apache.kafka.connect.runtime.distributed.DistributedHerder) [DistributedHerder-connect-1-1]
2023-06-05 15:06:39,498 INFO [Worker clientId=connect-1, groupId=connect-cluster] Rebalance started (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-06-05 15:06:39,498 INFO [Worker clientId=connect-1, groupId=connect-cluster] (Re-)joining group (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-06-05 15:06:39,507 INFO [Worker clientId=connect-1, groupId=connect-cluster] Successfully joined group with generation Generation{generationId=28890, memberId='evh-pars-dev-centralus.servicebus.windows.net:c:connect-cluster:I:connect-1-805b149901cf40948dfdab5f8f568065', protocol='sessioned'} (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-06-05 15:06:39,522 INFO [Worker clientId=connect-1, groupId=connect-cluster] SyncGroup failed: The group began another rebalance. Need to re-join the group. Sent generation was Generation{generationId=28890, memberId='evh-pars-dev-centralus.servicebus.windows.net:c:connect-cluster:I:connect-1-805b149901cf40948dfdab5f8f568065', protocol='sessioned'} (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-06-05 15:06:39,522 INFO [Worker clientId=connect-1, groupId=connect-cluster] Request joining group due to: rebalance failed due to 'The group is rebalancing, so a rejoin is needed.' (RebalanceInProgressException) (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-06-05 15:06:39,522 INFO [Worker clientId=connect-1, groupId=connect-cluster] (Re-)joining group (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-06-05 15:06:39,633 INFO [Worker clientId=connect-1, groupId=connect-cluster] Successfully joined group with generation Generation{generationId=28891, memberId='evh-pars-dev-centralus.servicebus.windows.net:c:connect-cluster:I:connect-1-805b149901cf40948dfdab5f8f568065', protocol='sessioned'} (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-06-05 15:06:39,647 INFO [Worker clientId=connect-1, groupId=connect-cluster] Successfully synced group in generation Generation{generationId=28891, memberId='evh-pars-dev-centralus.servicebus.windows.net:c:connect-cluster:I:connect-1-805b149901cf40948dfdab5f8f568065', protocol='sessioned'} (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-06-05 15:06:39,648 INFO [Worker clientId=connect-1, groupId=connect-cluster] Joined group at generation 28891 with protocol version 2 and got assignment: Assignment{error=1, leader='evh-pars-dev-centralus.servicebus.windows.net:c:connect-cluster:I:connect-1-f2ebe64c8c3a49b09a7db33b57713f81', leaderUrl='http://10.244.15.4:8083/', offset=669, connectorIds=[], taskIds=[], revokedConnectorIds=[], revokedTaskIds=[], delay=0} with rebalance delay: 0 (org.apache.kafka.connect.runtime.distributed.DistributedHerder) [DistributedHerder-connect-1-1]
2023-06-05 15:06:39,648 INFO [Worker clientId=connect-1, groupId=connect-cluster] Rebalance started (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-06-05 15:06:39,648 INFO [Worker clientId=connect-1, groupId=connect-cluster] (Re-)joining group (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-06-05 15:06:39,663 INFO [Worker clientId=connect-1, groupId=connect-cluster] Successfully joined group with generation Generation{generationId=28892, memberId='evh-pars-dev-centralus.servicebus.windows.net:c:connect-cluster:I:connect-1-805b149901cf40948dfdab5f8f568065', protocol='sessioned'} (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-06-05 15:06:39,679 INFO [Worker clientId=connect-1, groupId=connect-cluster] Successfully synced group in generation Generation{generationId=28892, memberId='evh-pars-dev-centralus.servicebus.windows.net:c:connect-cluster:I:connect-1-805b149901cf40948dfdab5f8f568065', protocol='sessioned'} (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-06-05 15:06:39,679 INFO [Worker clientId=connect-1, groupId=connect-cluster] Joined group at generation 28892 with protocol version 2 and got assignment: Assignment{error=1, leader='evh-pars-dev-centralus.servicebus.windows.net:c:connect-cluster:I:connect-1-f2ebe64c8c3a49b09a7db33b57713f81', leaderUrl='http://10.244.15.4:8083/', offset=669, connectorIds=[], taskIds=[], revokedConnectorIds=[], revokedTaskIds=[], delay=0} with rebalance delay: 0 (org.apache.kafka.connect.runtime.distributed.DistributedHerder) [DistributedHerder-connect-1-1]
2023-06-05 15:06:39,679 INFO [Worker clientId=connect-1, groupId=connect-cluster] Rebalance started (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-06-05 15:06:39,679 INFO [Worker clientId=connect-1, groupId=connect-cluster] (Re-)joining group (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-06-05 15:06:39,695 INFO [Worker clientId=connect-1, groupId=connect-cluster] Successfully joined group with generation Generation{generationId=28893, memberId='evh-pars-dev-centralus.servicebus.windows.net:c:connect-cluster:I:connect-1-805b149901cf40948dfdab5f8f568065', protocol='sessioned'} (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-06-05 15:06:39,710 INFO [Worker clientId=connect-1, groupId=connect-cluster] Successfully synced group in generation Generation{generationId=28893, memberId='evh-pars-dev-centralus.servicebus.windows.net:c:connect-cluster:I:connect-1-805b149901cf40948dfdab5f8f568065', protocol='sessioned'} (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-06-05 15:06:39,710 INFO [Worker clientId=connect-1, groupId=connect-cluster] Joined group at generation 28893 with protocol version 2 and got assignment: Assignment{error=1, leader='evh-pars-dev-centralus.servicebus.windows.net:c:connect-cluster:I:connect-1-f2ebe64c8c3a49b09a7db33b57713f81', leaderUrl='http://10.244.15.4:8083/', offset=669, connectorIds=[], taskIds=[], revokedConnectorIds=[], revokedTaskIds=[], delay=0} with rebalance delay: 0 (org.apache.kafka.connect.runtime.distributed.DistributedHerder) [DistributedHerder-connect-1-1]
2023-06-05 15:06:39,710 INFO [Worker clientId=connect-1, groupId=connect-cluster] Rebalance started (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-06-05 15:06:39,710 INFO [Worker clientId=connect-1, groupId=connect-cluster] (Re-)joining group (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-06-05 15:06:39,726 INFO [Worker clientId=connect-1, groupId=connect-cluster] Successfully joined group with generation Generation{generationId=28894, memberId='evh-pars-dev-centralus.servicebus.windows.net:c:connect-cluster:I:connect-1-805b149901cf40948dfdab5f8f568065', protocol='sessioned'} (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-06-05 15:06:39,741 INFO [Worker clientId=connect-1, groupId=connect-cluster] Successfully synced group in generation Generation{generationId=28894, memberId='evh-pars-dev-centralus.servicebus.windows.net:c:connect-cluster:I:connect-1-805b149901cf40948dfdab5f8f568065', protocol='sessioned'} (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-06-05 15:06:39,741 INFO [Worker clientId=connect-1, groupId=connect-cluster] Joined group at generation 28894 with protocol version 2 and got assignment: Assignment{error=1, leader='evh-pars-dev-centralus.servicebus.windows.net:c:connect-cluster:I:connect-1-f2ebe64c8c3a49b09a7db33b57713f81', leaderUrl='http://10.244.15.4:8083/', offset=669, connectorIds=[], taskIds=[], revokedConnectorIds=[], revokedTaskIds=[], delay=0} with rebalance delay: 0 (org.apache.kafka.connect.runtime.distributed.DistributedHerder) [DistributedHerder-connect-1-1]
2023-06-05 15:06:39,741 INFO [Worker clientId=connect-1, groupId=connect-cluster] Rebalance started (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-06-05 15:06:39,741 INFO [Worker clientId=connect-1, groupId=connect-cluster] (Re-)joining group (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-06-05 15:06:39,757 INFO [Worker clientId=connect-1, groupId=connect-cluster] Successfully joined group with generation Generation{generationId=28895, memberId='evh-pars-dev-centralus.servicebus.windows.net:c:connect-cluster:I:connect-1-805b149901cf40948dfdab5f8f568065', protocol='sessioned'} (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-06-05 15:06:39,772 INFO [Worker clientId=connect-1, groupId=connect-cluster] Successfully synced group in generation Generation{generationId=28895, memberId='evh-pars-dev-centralus.servicebus.windows.net:c:connect-cluster:I:connect-1-805b149901cf40948dfdab5f8f568065', protocol='sessioned'} (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-06-05 15:06:39,772 INFO [Worker clientId=connect-1, groupId=connect-cluster] Joined group at generation 28895 with protocol version 2 and got assignment: Assignment{error=1, leader='evh-pars-dev-centralus.servicebus.windows.net:c:connect-cluster:I:connect-1-f2ebe64c8c3a49b09a7db33b57713f81', leaderUrl='http://10.244.15.4:8083/', offset=669, connectorIds=[], taskIds=[], revokedConnectorIds=[], revokedTaskIds=[], delay=0} with rebalance delay: 0 (org.apache.kafka.connect.runtime.distributed.DistributedHerder) [DistributedHerder-connect-1-1]
2023-06-05 15:06:39,772 INFO [Worker clientId=connect-1, groupId=connect-cluster] Rebalance started (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-06-05 15:06:39,772 INFO [Worker clientId=connect-1, groupId=connect-cluster] (Re-)joining group (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-06-05 15:06:39,879 INFO [Worker clientId=connect-1, groupId=connect-cluster] Successfully joined group with generation Generation{generationId=28896, memberId='evh-pars-dev-centralus.servicebus.windows.net:c:connect-cluster:I:connect-1-805b149901cf40948dfdab5f8f568065', protocol='sessioned'} (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-06-05 15:06:39,889 INFO [Worker clientId=connect-1, groupId=connect-cluster] Successfully synced group in generation Generation{generationId=28896, memberId='evh-pars-dev-centralus.servicebus.windows.net:c:connect-cluster:I:connect-1-805b149901cf40948dfdab5f8f568065', protocol='sessioned'} (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-06-05 15:06:39,889 INFO [Worker clientId=connect-1, groupId=connect-cluster] Joined group at generation 28896 with protocol version 2 and got assignment: Assignment{error=1, leader='evh-pars-dev-centralus.servicebus.windows.net:c:connect-cluster:I:connect-1-f2ebe64c8c3a49b09a7db33b57713f81', leaderUrl='http://10.244.15.4:8083/', offset=669, connectorIds=[], taskIds=[], revokedConnectorIds=[], revokedTaskIds=[], delay=0} with rebalance delay: 0 (org.apache.kafka.connect.runtime.distributed.DistributedHerder) [DistributedHerder-connect-1-1]
2023-06-05 15:06:39,889 INFO [Worker clientId=connect-1, groupId=connect-cluster] Rebalance started (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-06-05 15:06:39,889 INFO [Worker clientId=connect-1, groupId=connect-cluster] (Re-)joining group (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-06-05 15:06:40,005 INFO [Worker clientId=connect-1, groupId=connect-cluster] Successfully joined group with generation Generation{generationId=28897, memberId='evh-pars-dev-centralus.servicebus.windows.net:c:connect-cluster:I:connect-1-805b149901cf40948dfdab5f8f568065', protocol='sessioned'} (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-06-05 15:06:40,023 INFO [Worker clientId=connect-1, groupId=connect-cluster] Successfully synced group in generation Generation{generationId=28897, memberId='evh-pars-dev-centralus.servicebus.windows.net:c:connect-cluster:I:connect-1-805b149901cf40948dfdab5f8f568065', protocol='sessioned'} (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-06-05 15:06:40,024 INFO [Worker clientId=connect-1, groupId=connect-cluster] Joined group at generation 28897 with protocol version 2 and got assignment: Assignment{error=1, leader='evh-pars-dev-centralus.servicebus.windows.net:c:connect-cluster:I:connect-1-f2ebe64c8c3a49b09a7db33b57713f81', leaderUrl='http://10.244.15.4:8083/', offset=669, connectorIds=[], taskIds=[], revokedConnectorIds=[], revokedTaskIds=[], delay=0} with rebalance delay: 0 (org.apache.kafka.connect.runtime.distributed.DistributedHerder) [DistributedHerder-connect-1-1]
2023-06-05 15:06:40,024 INFO [Worker clientId=connect-1, groupId=connect-cluster] Rebalance started (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-06-05 15:06:40,024 INFO [Worker clientId=connect-1, groupId=connect-cluster] (Re-)joining group (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-06-05 15:06:40,129 INFO [Worker clientId=connect-1, groupId=connect-cluster] Successfully joined group with generation Generation{generationId=28897, memberId='evh-pars-dev-centralus.servicebus.windows.net:c:connect-cluster:I:connect-1-805b149901cf40948dfdab5f8f568065', protocol='sessioned'} (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-06-05 15:06:40,132 INFO [Worker clientId=connect-1, groupId=connect-cluster] SyncGroup failed: The group began another rebalance. Need to re-join the group. Sent generation was Generation{generationId=28897, memberId='evh-pars-dev-centralus.servicebus.windows.net:c:connect-cluster:I:connect-1-805b149901cf40948dfdab5f8f568065', protocol='sessioned'} (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-06-05 15:06:40,132 INFO [Worker clientId=connect-1, groupId=connect-cluster] Request joining group due to: rebalance failed due to 'The group is rebalancing, so a rejoin is needed.' (RebalanceInProgressException) (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-06-05 15:06:40,132 INFO [Worker clientId=connect-1, groupId=connect-cluster] (Re-)joining group (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-06-05 15:06:40,148 INFO [Worker clientId=connect-1, groupId=connect-cluster] Successfully joined group with generation Generation{generationId=28898, memberId='evh-pars-dev-centralus.servicebus.windows.net:c:connect-cluster:I:connect-1-805b149901cf40948dfdab5f8f568065', protocol='sessioned'} (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-06-05 15:06:40,163 INFO [Worker clientId=connect-1, groupId=connect-cluster] Successfully synced group in generation Generation{generationId=28898, memberId='evh-pars-dev-centralus.servicebus.windows.net:c:connect-cluster:I:connect-1-805b149901cf40948dfdab5f8f568065', protocol='sessioned'} (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-06-05 15:06:40,163 INFO [Worker clientId=connect-1, groupId=connect-cluster] Joined group at generation 28898 with protocol version 2 and got assignment: Assignment{error=1, leader='evh-pars-dev-centralus.servicebus.windows.net:c:connect-cluster:I:connect-1-f2ebe64c8c3a49b09a7db33b57713f81', leaderUrl='http://10.244.15.4:8083/', offset=669, connectorIds=[], taskIds=[], revokedConnectorIds=[], revokedTaskIds=[], delay=0} with rebalance delay: 0 (org.apache.kafka.connect.runtime.distributed.DistributedHerder) [DistributedHerder-connect-1-1]
2023-06-05 15:06:40,163 INFO [Worker clientId=connect-1, groupId=connect-cluster] Rebalance started (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-06-05 15:06:40,163 INFO [Worker clientId=connect-1, groupId=connect-cluster] (Re-)joining group (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-06-05 15:06:40,270 INFO [Worker clientId=connect-1, groupId=connect-cluster] Successfully joined group with generation Generation{generationId=28898, memberId='evh-pars-dev-centralus.servicebus.windows.net:c:connect-cluster:I:connect-1-805b149901cf40948dfdab5f8f568065', protocol='sessioned'} (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-06-05 15:06:40,272 INFO [Worker clientId=connect-1, groupId=connect-cluster] SyncGroup failed: The group began another rebalance. Need to re-join the group. Sent generation was Generation{generationId=28898, memberId='evh-pars-dev-centralus.servicebus.windows.net:c:connect-cluster:I:connect-1-805b149901cf40948dfdab5f8f568065', protocol='sessioned'} (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-06-05 15:06:40,272 INFO [Worker clientId=connect-1, groupId=connect-cluster] Request joining group due to: rebalance failed due to 'The group is rebalancing, so a rejoin is needed.' (RebalanceInProgressException) (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-06-05 15:06:40,272 INFO [Worker clientId=connect-1, groupId=connect-cluster] (Re-)joining group (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-06-05 15:06:40,289 INFO [Worker clientId=connect-1, groupId=connect-cluster] Successfully joined group with generation Generation{generationId=28899, memberId='evh-pars-dev-centralus.servicebus.windows.net:c:connect-cluster:I:connect-1-805b149901cf40948dfdab5f8f568065', protocol='sessioned'} (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-06-05 15:06:40,304 INFO [Worker clientId=connect-1, groupId=connect-cluster] Successfully synced group in generation Generation{generationId=28899, memberId='evh-pars-dev-centralus.servicebus.windows.net:c:connect-cluster:I:connect-1-805b149901cf40948dfdab5f8f568065', protocol='sessioned'} (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-06-05 15:06:40,304 INFO [Worker clientId=connect-1, groupId=connect-cluster] Joined group at generation 28899 with protocol version 2 and got assignment: Assignment{error=1, leader='evh-pars-dev-centralus.servicebus.windows.net:c:connect-cluster:I:connect-1-f2ebe64c8c3a49b09a7db33b57713f81', leaderUrl='http://10.244.15.4:8083/', offset=669, connectorIds=[], taskIds=[], revokedConnectorIds=[], revokedTaskIds=[], delay=0} with rebalance delay: 0 (org.apache.kafka.connect.runtime.distributed.DistributedHerder) [DistributedHerder-connect-1-1]
2023-06-05 15:06:40,304 INFO [Worker clientId=connect-1, groupId=connect-cluster] Rebalance started (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-06-05 15:06:40,304 INFO [Worker clientId=connect-1, groupId=connect-cluster] (Re-)joining group (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-06-05 15:06:40,319 INFO [Worker clientId=connect-1, groupId=connect-cluster] Successfully joined group with generation Generation{generationId=28899, memberId='evh-pars-dev-centralus.servicebus.windows.net:c:connect-cluster:I:connect-1-805b149901cf40948dfdab5f8f568065', protocol='sessioned'} (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-06-05 15:06:40,335 INFO [Worker clientId=connect-1, groupId=connect-cluster] Successfully synced group in generation Generation{generationId=28899, memberId='evh-pars-dev-centralus.servicebus.windows.net:c:connect-cluster:I:connect-1-805b149901cf40948dfdab5f8f568065', protocol='sessioned'} (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-06-05 15:06:40,335 INFO [Worker clientId=connect-1, groupId=connect-cluster] Joined group at generation 28899 with protocol version 2 and got assignment: Assignment{error=1, leader='evh-pars-dev-centralus.servicebus.windows.net:c:connect-cluster:I:connect-1-f2ebe64c8c3a49b09a7db33b57713f81', leaderUrl='http://10.244.15.4:8083/', offset=669, connectorIds=[], taskIds=[], revokedConnectorIds=[], revokedTaskIds=[], delay=0} with rebalance delay: 0 (org.apache.kafka.connect.runtime.distributed.DistributedHerder) [DistributedHerder-connect-1-1]
2023-06-05 15:06:40,335 INFO [Worker clientId=connect-1, groupId=connect-cluster] Rebalance started (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-06-05 15:06:40,335 INFO [Worker clientId=connect-1, groupId=connect-cluster] (Re-)joining group (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-06-05 15:06:40,351 INFO [Worker clientId=connect-1, groupId=connect-cluster] Successfully joined group with generation Generation{generationId=28899, memberId='evh-pars-dev-centralus.servicebus.windows.net:c:connect-cluster:I:connect-1-805b149901cf40948dfdab5f8f568065', protocol='sessioned'} (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-06-05 15:06:40,366 INFO [Worker clientId=connect-1, groupId=connect-cluster] SyncGroup failed: The group began another rebalance. Need to re-join the group. Sent generation was Generation{generationId=28899, memberId='evh-pars-dev-centralus.servicebus.windows.net:c:connect-cluster:I:connect-1-805b149901cf40948dfdab5f8f568065', protocol='sessioned'} (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-06-05 15:06:40,366 INFO [Worker clientId=connect-1, groupId=connect-cluster] Request joining group due to: rebalance failed due to 'The group is rebalancing, so a rejoin is needed.' (RebalanceInProgressException) (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-06-05 15:06:40,366 INFO [Worker clientId=connect-1, groupId=connect-cluster] (Re-)joining group (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-06-05 15:06:40,473 INFO [Worker clientId=connect-1, groupId=connect-cluster] Successfully joined group with generation Generation{generationId=28900, memberId='evh-pars-dev-centralus.servicebus.windows.net:c:connect-cluster:I:connect-1-805b149901cf40948dfdab5f8f568065', protocol='sessioned'} (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-06-05 15:06:40,491 INFO [Worker clientId=connect-1, groupId=connect-cluster] Successfully synced group in generation Generation{generationId=28900, memberId='evh-pars-dev-centralus.servicebus.windows.net:c:connect-cluster:I:connect-1-805b149901cf40948dfdab5f8f568065', protocol='sessioned'} (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-06-05 15:06:40,491 INFO [Worker clientId=connect-1, groupId=connect-cluster] Joined group at generation 28900 with protocol version 2 and got assignment: Assignment{error=1, leader='evh-pars-dev-centralus.servicebus.windows.net:c:connect-cluster:I:connect-1-f2ebe64c8c3a49b09a7db33b57713f81', leaderUrl='http://10.244.15.4:8083/', offset=669, connectorIds=[], taskIds=[], revokedConnectorIds=[], revokedTaskIds=[], delay=0} with rebalance delay: 0 (org.apache.kafka.connect.runtime.distributed.DistributedHerder) [DistributedHerder-connect-1-1]
2023-06-05 15:06:40,491 INFO [Worker clientId=connect-1, groupId=connect-cluster] Rebalance started (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-06-05 15:06:40,491 INFO [Worker clientId=connect-1, groupId=connect-cluster] (Re-)joining group (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-06-05 15:06:40,507 INFO [Worker clientId=connect-1, groupId=connect-cluster] Successfully joined group with generation Generation{generationId=28901, memberId='evh-pars-dev-centralus.servicebus.windows.net:c:connect-cluster:I:connect-1-805b149901cf40948dfdab5f8f568065', protocol='sessioned'} (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-06-05 15:06:40,522 INFO [Worker clientId=connect-1, groupId=connect-cluster] Successfully synced group in generation Generation{generationId=28901, memberId='evh-pars-dev-centralus.servicebus.windows.net:c:connect-cluster:I:connect-1-805b149901cf40948dfdab5f8f568065', protocol='sessioned'} (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-06-05 15:06:40,522 INFO [Worker clientId=connect-1, groupId=connect-cluster] Joined group at generation 28901 with protocol version 2 and got assignment: Assignment{error=1, leader='evh-pars-dev-centralus.servicebus.windows.net:c:connect-cluster:I:connect-1-f2ebe64c8c3a49b09a7db33b57713f81', leaderUrl='http://10.244.15.4:8083/', offset=669, connectorIds=[], taskIds=[], revokedConnectorIds=[], revokedTaskIds=[], delay=0} with rebalance delay: 0 (org.apache.kafka.connect.runtime.distributed.DistributedHerder) [DistributedHerder-connect-1-1]
2023-06-05 15:06:40,522 INFO [Worker clientId=connect-1, groupId=connect-cluster] Rebalance started (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-06-05 15:06:40,522 INFO [Worker clientId=connect-1, groupId=connect-cluster] (Re-)joining group (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-06-05 15:06:40,629 INFO [Worker clientId=connect-1, groupId=connect-cluster] Successfully joined group with generation Generation{generationId=28901, memberId='evh-pars-dev-centralus.servicebus.windows.net:c:connect-cluster:I:connect-1-805b149901cf40948dfdab5f8f568065', protocol='sessioned'} (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-06-05 15:06:40,632 INFO [Worker clientId=connect-1, groupId=connect-cluster] SyncGroup failed: The group began another rebalance. Need to re-join the group. Sent generation was Generation{generationId=28901, memberId='evh-pars-dev-centralus.servicebus.windows.net:c:connect-cluster:I:connect-1-805b149901cf40948dfdab5f8f568065', protocol='sessioned'} (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-06-05 15:06:40,632 INFO [Worker clientId=connect-1, groupId=connect-cluster] Request joining group due to: rebalance failed due to 'The group is rebalancing, so a rejoin is needed.' (RebalanceInProgressException) (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-06-05 15:06:40,632 INFO [Worker clientId=connect-1, groupId=connect-cluster] (Re-)joining group (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-06-05 15:06:40,739 INFO [Worker clientId=connect-1, groupId=connect-cluster] Successfully joined group with generation Generation{generationId=28902, memberId='evh-pars-dev-centralus.servicebus.windows.net:c:connect-cluster:I:connect-1-805b149901cf40948dfdab5f8f568065', protocol='sessioned'} (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-06-05 15:06:40,757 INFO [Worker clientId=connect-1, groupId=connect-cluster] Successfully synced group in generation Generation{generationId=28902, memberId='evh-pars-dev-centralus.servicebus.windows.net:c:connect-cluster:I:connect-1-805b149901cf40948dfdab5f8f568065', protocol='sessioned'} (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-06-05 15:06:40,757 INFO [Worker clientId=connect-1, groupId=connect-cluster] Joined group at generation 28902 with protocol version 2 and got assignment: Assignment{error=1, leader='evh-pars-dev-centralus.servicebus.windows.net:c:connect-cluster:I:connect-1-f2ebe64c8c3a49b09a7db33b57713f81', leaderUrl='http://10.244.15.4:8083/', offset=669, connectorIds=[], taskIds=[], revokedConnectorIds=[], revokedTaskIds=[], delay=0} with rebalance delay: 0 (org.apache.kafka.connect.runtime.distributed.DistributedHerder) [DistributedHerder-connect-1-1]
2023-06-05 15:06:40,757 INFO [Worker clientId=connect-1, groupId=connect-cluster] Rebalance started (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-06-05 15:06:40,757 INFO [Worker clientId=connect-1, groupId=connect-cluster] (Re-)joining group (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-06-05 15:06:40,773 INFO [Worker clientId=connect-1, groupId=connect-cluster] Successfully joined group with generation Generation{generationId=28903, memberId='evh-pars-dev-centralus.servicebus.windows.net:c:connect-cluster:I:connect-1-805b149901cf40948dfdab5f8f568065', protocol='sessioned'} (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-06-05 15:06:40,788 INFO [Worker clientId=connect-1, groupId=connect-cluster] Successfully synced group in generation Generation{generationId=28903, memberId='evh-pars-dev-centralus.servicebus.windows.net:c:connect-cluster:I:connect-1-805b149901cf40948dfdab5f8f568065', protocol='sessioned'} (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-06-05 15:06:40,788 INFO [Worker clientId=connect-1, groupId=connect-cluster] Joined group at generation 28903 with protocol version 2 and got assignment: Assignment{error=1, leader='evh-pars-dev-centralus.servicebus.windows.net:c:connect-cluster:I:connect-1-f2ebe64c8c3a49b09a7db33b57713f81', leaderUrl='http://10.244.15.4:8083/', offset=669, connectorIds=[], taskIds=[], revokedConnectorIds=[], revokedTaskIds=[], delay=0} with rebalance delay: 0 (org.apache.kafka.connect.runtime.distributed.DistributedHerder) [DistributedHerder-connect-1-1]
2023-06-05 15:06:40,788 INFO [Worker clientId=connect-1, groupId=connect-cluster] Rebalance started (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-06-05 15:06:40,788 INFO [Worker clientId=connect-1, groupId=connect-cluster] (Re-)joining group (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-06-05 15:06:40,804 INFO [Worker clientId=connect-1, groupId=connect-cluster] Successfully joined group with generation Generation{generationId=28904, memberId='evh-pars-dev-centralus.servicebus.windows.net:c:connect-cluster:I:connect-1-805b149901cf40948dfdab5f8f568065', protocol='sessioned'} (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-06-05 15:06:40,819 INFO [Worker clientId=connect-1, groupId=connect-cluster] Successfully synced group in generation Generation{generationId=28904, memberId='evh-pars-dev-centralus.servicebus.windows.net:c:connect-cluster:I:connect-1-805b149901cf40948dfdab5f8f568065', protocol='sessioned'} (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-06-05 15:06:40,819 INFO [Worker clientId=connect-1, groupId=connect-cluster] Joined group at generation 28904 with protocol version 2 and got assignment: Assignment{error=1, leader='evh-pars-dev-centralus.servicebus.windows.net:c:connect-cluster:I:connect-1-f2ebe64c8c3a49b09a7db33b57713f81', leaderUrl='http://10.244.15.4:8083/', offset=669, connectorIds=[], taskIds=[], revokedConnectorIds=[], revokedTaskIds=[], delay=0} with rebalance delay: 0 (org.apache.kafka.connect.runtime.distributed.DistributedHerder) [DistributedHerder-connect-1-1]
2023-06-05 15:06:40,819 INFO [Worker clientId=connect-1, groupId=connect-cluster] Rebalance started (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-06-05 15:06:40,819 INFO [Worker clientId=connect-1, groupId=connect-cluster] (Re-)joining group (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-06-05 15:06:40,926 INFO [Worker clientId=connect-1, groupId=connect-cluster] Successfully joined group with generation Generation{generationId=28904, memberId='evh-pars-dev-centralus.servicebus.windows.net:c:connect-cluster:I:connect-1-805b149901cf40948dfdab5f8f568065', protocol='sessioned'} (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-06-05 15:06:40,929 INFO [Worker clientId=connect-1, groupId=connect-cluster] SyncGroup failed: The group began another rebalance. Need to re-join the group. Sent generation was Generation{generationId=28904, memberId='evh-pars-dev-centralus.servicebus.windows.net:c:connect-cluster:I:connect-1-805b149901cf40948dfdab5f8f568065', protocol='sessioned'} (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-06-05 15:06:40,929 INFO [Worker clientId=connect-1, groupId=connect-cluster] Request joining group due to: rebalance failed due to 'The group is rebalancing, so a rejoin is needed.' (RebalanceInProgressException) (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-06-05 15:06:40,929 INFO [Worker clientId=connect-1, groupId=connect-cluster] (Re-)joining group (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-06-05 15:06:40,945 INFO [Worker clientId=connect-1, groupId=connect-cluster] Successfully joined group with generation Generation{generationId=28905, memberId='evh-pars-dev-centralus.servicebus.windows.net:c:connect-cluster:I:connect-1-805b149901cf40948dfdab5f8f568065', protocol='sessioned'} (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-06-05 15:06:40,960 INFO [Worker clientId=connect-1, groupId=connect-cluster] Successfully synced group in generation Generation{generationId=28905, memberId='evh-pars-dev-centralus.servicebus.windows.net:c:connect-cluster:I:connect-1-805b149901cf40948dfdab5f8f568065', protocol='sessioned'} (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-06-05 15:06:40,960 INFO [Worker clientId=connect-1, groupId=connect-cluster] Joined group at generation 28905 with protocol version 2 and got assignment: Assignment{error=1, leader='evh-pars-dev-centralus.servicebus.windows.net:c:connect-cluster:I:connect-1-f2ebe64c8c3a49b09a7db33b57713f81', leaderUrl='http://10.244.15.4:8083/', offset=669, connectorIds=[], taskIds=[], revokedConnectorIds=[], revokedTaskIds=[], delay=0} with rebalance delay: 0 (org.apache.kafka.connect.runtime.distributed.DistributedHerder) [DistributedHerder-connect-1-1]
2023-06-05 15:06:40,960 INFO [Worker clientId=connect-1, groupId=connect-cluster] Rebalance started (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-06-05 15:06:40,960 INFO [Worker clientId=connect-1, groupId=connect-cluster] (Re-)joining group (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-06-05 15:06:40,976 INFO [Worker clientId=connect-1, groupId=connect-cluster] Successfully joined group with generation Generation{generationId=28906, memberId='evh-pars-dev-centralus.servicebus.windows.net:c:connect-cluster:I:connect-1-805b149901cf40948dfdab5f8f568065', protocol='sessioned'} (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-06-05 15:06:40,991 INFO [Worker clientId=connect-1, groupId=connect-cluster] Successfully synced group in generation Generation{generationId=28906, memberId='evh-pars-dev-centralus.servicebus.windows.net:c:connect-cluster:I:connect-1-805b149901cf40948dfdab5f8f568065', protocol='sessioned'} (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-06-05 15:06:40,991 INFO [Worker clientId=connect-1, groupId=connect-cluster] Joined group at generation 28906 with protocol version 2 and got assignment: Assignment{error=1, leader='evh-pars-dev-centralus.servicebus.windows.net:c:connect-cluster:I:connect-1-f2ebe64c8c3a49b09a7db33b57713f81', leaderUrl='http://10.244.15.4:8083/', offset=669, connectorIds=[], taskIds=[], revokedConnectorIds=[], revokedTaskIds=[], delay=0} with rebalance delay: 0 (org.apache.kafka.connect.runtime.distributed.DistributedHerder) [DistributedHerder-connect-1-1]
2023-06-05 15:06:40,991 INFO [Worker clientId=connect-1, groupId=connect-cluster] Rebalance started (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-06-05 15:06:40,991 INFO [Worker clientId=connect-1, groupId=connect-cluster] (Re-)joining group (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-06-05 15:06:41,007 INFO [Worker clientId=connect-1, groupId=connect-cluster] Successfully joined group with generation Generation{generationId=28907, memberId='evh-pars-dev-centralus.servicebus.windows.net:c:connect-cluster:I:connect-1-805b149901cf40948dfdab5f8f568065', protocol='sessioned'} (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-06-05 15:06:41,022 INFO [Worker clientId=connect-1, groupId=connect-cluster] Successfully synced group in generation Generation{generationId=28907, memberId='evh-pars-dev-centralus.servicebus.windows.net:c:connect-cluster:I:connect-1-805b149901cf40948dfdab5f8f568065', protocol='sessioned'} (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-06-05 15:06:41,022 INFO [Worker clientId=connect-1, groupId=connect-cluster] Joined group at generation 28907 with protocol version 2 and got assignment: Assignment{error=1, leader='evh-pars-dev-centralus.servicebus.windows.net:c:connect-cluster:I:connect-1-f2ebe64c8c3a49b09a7db33b57713f81', leaderUrl='http://10.244.15.4:8083/', offset=669, connectorIds=[], taskIds=[], revokedConnectorIds=[], revokedTaskIds=[], delay=0} with rebalance delay: 0 (org.apache.kafka.connect.runtime.distributed.DistributedHerder) [DistributedHerder-connect-1-1]
2023-06-05 15:06:41,022 INFO [Worker clientId=connect-1, groupId=connect-cluster] Rebalance started (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-06-05 15:06:41,022 INFO [Worker clientId=connect-1, groupId=connect-cluster] (Re-)joining group (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-06-05 15:06:41,038 INFO [Worker clientId=connect-1, groupId=connect-cluster] Successfully joined group with generation Generation{generationId=28908, memberId='evh-pars-dev-centralus.servicebus.windows.net:c:connect-cluster:I:connect-1-805b149901cf40948dfdab5f8f568065', protocol='sessioned'} (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-06-05 15:06:41,053 INFO [Worker clientId=connect-1, groupId=connect-cluster] Successfully synced group in generation Generation{generationId=28908, memberId='evh-pars-dev-centralus.servicebus.windows.net:c:connect-cluster:I:connect-1-805b149901cf40948dfdab5f8f568065', protocol='sessioned'} (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-06-05 15:06:41,054 INFO [Worker clientId=connect-1, groupId=connect-cluster] Joined group at generation 28908 with protocol version 2 and got assignment: Assignment{error=1, leader='evh-pars-dev-centralus.servicebus.windows.net:c:connect-cluster:I:connect-1-f2ebe64c8c3a49b09a7db33b57713f81', leaderUrl='http://10.244.15.4:8083/', offset=669, connectorIds=[], taskIds=[], revokedConnectorIds=[], revokedTaskIds=[], delay=0} with rebalance delay: 0 (org.apache.kafka.connect.runtime.distributed.DistributedHerder) [DistributedHerder-connect-1-1]
2023-06-05 15:06:41,054 INFO [Worker clientId=connect-1, groupId=connect-cluster] Rebalance started (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-06-05 15:06:41,054 INFO [Worker clientId=connect-1, groupId=connect-cluster] (Re-)joining group (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-06-05 15:06:41,069 INFO [Worker clientId=connect-1, groupId=connect-cluster] Successfully joined group with generation Generation{generationId=28909, memberId='evh-pars-dev-centralus.servicebus.windows.net:c:connect-cluster:I:connect-1-805b149901cf40948dfdab5f8f568065', protocol='sessioned'} (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-06-05 15:06:41,085 INFO [Worker clientId=connect-1, groupId=connect-cluster] Successfully synced group in generation Generation{generationId=28909, memberId='evh-pars-dev-centralus.servicebus.windows.net:c:connect-cluster:I:connect-1-805b149901cf40948dfdab5f8f568065', protocol='sessioned'} (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-06-05 15:06:41,085 INFO [Worker clientId=connect-1, groupId=connect-cluster] Joined group at generation 28909 with protocol version 2 and got assignment: Assignment{error=1, leader='evh-pars-dev-centralus.servicebus.windows.net:c:connect-cluster:I:connect-1-f2ebe64c8c3a49b09a7db33b57713f81', leaderUrl='http://10.244.15.4:8083/', offset=669, connectorIds=[], taskIds=[], revokedConnectorIds=[], revokedTaskIds=[], delay=0} with rebalance delay: 0 (org.apache.kafka.connect.runtime.distributed.DistributedHerder) [DistributedHerder-connect-1-1]
2023-06-05 15:06:41,085 INFO [Worker clientId=connect-1, groupId=connect-cluster] Rebalance started (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-06-05 15:06:41,085 INFO [Worker clientId=connect-1, groupId=connect-cluster] (Re-)joining group (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-06-05 15:06:41,192 INFO [Worker clientId=connect-1, groupId=connect-cluster] Successfully joined group with generation Generation{generationId=28910, memberId='evh-pars-dev-centralus.servicebus.windows.net:c:connect-cluster:I:connect-1-805b149901cf40948dfdab5f8f568065', protocol='sessioned'} (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-06-05 15:06:41,210 INFO [Worker clientId=connect-1, groupId=connect-cluster] Successfully synced group in generation Generation{generationId=28910, memberId='evh-pars-dev-centralus.servicebus.windows.net:c:connect-cluster:I:connect-1-805b149901cf40948dfdab5f8f568065', protocol='sessioned'} (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-06-05 15:06:41,210 INFO [Worker clientId=connect-1, groupId=connect-cluster] Joined group at generation 28910 with protocol version 2 and got assignment: Assignment{error=1, leader='evh-pars-dev-centralus.servicebus.windows.net:c:connect-cluster:I:connect-1-f2ebe64c8c3a49b09a7db33b57713f81', leaderUrl='http://10.244.15.4:8083/', offset=669, connectorIds=[], taskIds=[], revokedConnectorIds=[], revokedTaskIds=[], delay=0} with rebalance delay: 0 (org.apache.kafka.connect.runtime.distributed.DistributedHerder) [DistributedHerder-connect-1-1]
2023-06-05 15:06:41,210 INFO [Worker clientId=connect-1, groupId=connect-cluster] Rebalance started (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-06-05 15:06:41,210 INFO [Worker clientId=connect-1, groupId=connect-cluster] (Re-)joining group (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-06-05 15:06:41,226 INFO [Worker clientId=connect-1, groupId=connect-cluster] Successfully joined group with generation Generation{generationId=28911, memberId='evh-pars-dev-centralus.servicebus.windows.net:c:connect-cluster:I:connect-1-805b149901cf40948dfdab5f8f568065', protocol='sessioned'} (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-06-05 15:06:41,241 INFO [Worker clientId=connect-1, groupId=connect-cluster] Successfully synced group in generation Generation{generationId=28911, memberId='evh-pars-dev-centralus.servicebus.windows.net:c:connect-cluster:I:connect-1-805b149901cf40948dfdab5f8f568065', protocol='sessioned'} (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-06-05 15:06:41,241 INFO [Worker clientId=connect-1, groupId=connect-cluster] Joined group at generation 28911 with protocol version 2 and got assignment: Assignment{error=1, leader='evh-pars-dev-centralus.servicebus.windows.net:c:connect-cluster:I:connect-1-f2ebe64c8c3a49b09a7db33b57713f81', leaderUrl='http://10.244.15.4:8083/', offset=669, connectorIds=[], taskIds=[], revokedConnectorIds=[], revokedTaskIds=[], delay=0} with rebalance delay: 0 (org.apache.kafka.connect.runtime.distributed.DistributedHerder) [DistributedHerder-connect-1-1]
2023-06-05 15:06:41,241 INFO [Worker clientId=connect-1, groupId=connect-cluster] Rebalance started (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-06-05 15:06:41,241 INFO [Worker clientId=connect-1, groupId=connect-cluster] (Re-)joining group (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-06-05 15:06:41,257 INFO [Worker clientId=connect-1, groupId=connect-cluster] Successfully joined group with generation Generation{generationId=28912, memberId='evh-pars-dev-centralus.servicebus.windows.net:c:connect-cluster:I:connect-1-805b149901cf40948dfdab5f8f568065', protocol='sessioned'} (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-06-05 15:06:41,272 INFO [Worker clientId=connect-1, groupId=connect-cluster] Successfully synced group in generation Generation{generationId=28912, memberId='evh-pars-dev-centralus.servicebus.windows.net:c:connect-cluster:I:connect-1-805b149901cf40948dfdab5f8f568065', protocol='sessioned'} (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-06-05 15:06:41,272 INFO [Worker clientId=connect-1, groupId=connect-cluster] Joined group at generation 28912 with protocol version 2 and got assignment: Assignment{error=1, leader='evh-pars-dev-centralus.servicebus.windows.net:c:connect-cluster:I:connect-1-f2ebe64c8c3a49b09a7db33b57713f81', leaderUrl='http://10.244.15.4:8083/', offset=669, connectorIds=[], taskIds=[], revokedConnectorIds=[], revokedTaskIds=[], delay=0} with rebalance delay: 0 (org.apache.kafka.connect.runtime.distributed.DistributedHerder) [DistributedHerder-connect-1-1]
2023-06-05 15:06:41,272 INFO [Worker clientId=connect-1, groupId=connect-cluster] Rebalance started (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-06-05 15:06:41,272 INFO [Worker clientId=connect-1, groupId=connect-cluster] (Re-)joining group (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-06-05 15:06:41,288 INFO [Worker clientId=connect-1, groupId=connect-cluster] Successfully joined group with generation Generation{generationId=28913, memberId='evh-pars-dev-centralus.servicebus.windows.net:c:connect-cluster:I:connect-1-805b149901cf40948dfdab5f8f568065', protocol='sessioned'} (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-06-05 15:06:41,304 INFO [Worker clientId=connect-1, groupId=connect-cluster] Successfully synced group in generation Generation{generationId=28913, memberId='evh-pars-dev-centralus.servicebus.windows.net:c:connect-cluster:I:connect-1-805b149901cf40948dfdab5f8f568065', protocol='sessioned'} (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-06-05 15:06:41,304 INFO [Worker clientId=connect-1, groupId=connect-cluster] Joined group at generation 28913 with protocol version 2 and got assignment: Assignment{error=1, leader='evh-pars-dev-centralus.servicebus.windows.net:c:connect-cluster:I:connect-1-f2ebe64c8c3a49b09a7db33b57713f81', leaderUrl='http://10.244.15.4:8083/', offset=669, connectorIds=[], taskIds=[], revokedConnectorIds=[], revokedTaskIds=[], delay=0} with rebalance delay: 0 (org.apache.kafka.connect.runtime.distributed.DistributedHerder) [DistributedHerder-connect-1-1]
2023-06-05 15:06:41,304 INFO [Worker clientId=connect-1, groupId=connect-cluster] Rebalance started (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-06-05 15:06:41,304 INFO [Worker clientId=connect-1, groupId=connect-cluster] (Re-)joining group (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-06-05 15:06:41,319 INFO [Worker clientId=connect-1, groupId=connect-cluster] Successfully joined group with generation Generation{generationId=28914, memberId='evh-pars-dev-centralus.servicebus.windows.net:c:connect-cluster:I:connect-1-805b149901cf40948dfdab5f8f568065', protocol='sessioned'} (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-06-05 15:06:41,335 INFO [Worker clientId=connect-1, groupId=connect-cluster] Successfully synced group in generation Generation{generationId=28914, memberId='evh-pars-dev-centralus.servicebus.windows.net:c:connect-cluster:I:connect-1-805b149901cf40948dfdab5f8f568065', protocol='sessioned'} (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-06-05 15:06:41,335 INFO [Worker clientId=connect-1, groupId=connect-cluster] Joined group at generation 28914 with protocol version 2 and got assignment: Assignment{error=1, leader='evh-pars-dev-centralus.servicebus.windows.net:c:connect-cluster:I:connect-1-f2ebe64c8c3a49b09a7db33b57713f81', leaderUrl='http://10.244.15.4:8083/', offset=669, connectorIds=[], taskIds=[], revokedConnectorIds=[], revokedTaskIds=[], delay=0} with rebalance delay: 0 (org.apache.kafka.connect.runtime.distributed.DistributedHerder) [DistributedHerder-connect-1-1]
2023-06-05 15:06:41,335 INFO [Worker clientId=connect-1, groupId=connect-cluster] Rebalance started (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-06-05 15:06:41,335 INFO [Worker clientId=connect-1, groupId=connect-cluster] (Re-)joining group (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-06-05 15:06:41,442 INFO [Worker clientId=connect-1, groupId=connect-cluster] Successfully joined group with generation Generation{generationId=28914, memberId='evh-pars-dev-centralus.servicebus.windows.net:c:connect-cluster:I:connect-1-805b149901cf40948dfdab5f8f568065', protocol='sessioned'} (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-06-05 15:06:41,444 INFO [Worker clientId=connect-1, groupId=connect-cluster] SyncGroup failed: The group began another rebalance. Need to re-join the group. Sent generation was Generation{generationId=28914, memberId='evh-pars-dev-centralus.servicebus.windows.net:c:connect-cluster:I:connect-1-805b149901cf40948dfdab5f8f568065', protocol='sessioned'} (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-06-05 15:06:41,444 INFO [Worker clientId=connect-1, groupId=connect-cluster] Request joining group due to: rebalance failed due to 'The group is rebalancing, so a rejoin is needed.' (RebalanceInProgressException) (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-06-05 15:06:41,444 INFO [Worker clientId=connect-1, groupId=connect-cluster] (Re-)joining group (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-06-05 15:06:41,460 INFO [Worker clientId=connect-1, groupId=connect-cluster] Successfully joined group with generation Generation{generationId=28915, memberId='evh-pars-dev-centralus.servicebus.windows.net:c:connect-cluster:I:connect-1-805b149901cf40948dfdab5f8f568065', protocol='sessioned'} (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-06-05 15:06:41,476 INFO [Worker clientId=connect-1, groupId=connect-cluster] Successfully synced group in generation Generation{generationId=28915, memberId='evh-pars-dev-centralus.servicebus.windows.net:c:connect-cluster:I:connect-1-805b149901cf40948dfdab5f8f568065', protocol='sessioned'} (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-06-05 15:06:41,476 INFO [Worker clientId=connect-1, groupId=connect-cluster] Joined group at generation 28915 with protocol version 2 and got assignment: Assignment{error=1, leader='evh-pars-dev-centralus.servicebus.windows.net:c:connect-cluster:I:connect-1-f2ebe64c8c3a49b09a7db33b57713f81', leaderUrl='http://10.244.15.4:8083/', offset=669, connectorIds=[], taskIds=[], revokedConnectorIds=[], revokedTaskIds=[], delay=0} with rebalance delay: 0 (org.apache.kafka.connect.runtime.distributed.DistributedHerder) [DistributedHerder-connect-1-1]
2023-06-05 15:06:41,476 INFO [Worker clientId=connect-1, groupId=connect-cluster] Rebalance started (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-06-05 15:06:41,476 INFO [Worker clientId=connect-1, groupId=connect-cluster] (Re-)joining group (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-06-05 15:06:41,582 INFO [Worker clientId=connect-1, groupId=connect-cluster] Successfully joined group with generation Generation{generationId=28916, memberId='evh-pars-dev-centralus.servicebus.windows.net:c:connect-cluster:I:connect-1-805b149901cf40948dfdab5f8f568065', protocol='sessioned'} (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-06-05 15:06:41,601 INFO [Worker clientId=connect-1, groupId=connect-cluster] Successfully synced group in generation Generation{generationId=28916, memberId='evh-pars-dev-centralus.servicebus.windows.net:c:connect-cluster:I:connect-1-805b149901cf40948dfdab5f8f568065', protocol='sessioned'} (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-06-05 15:06:41,601 INFO [Worker clientId=connect-1, groupId=connect-cluster] Joined group at generation 28916 with protocol version 2 and got assignment: Assignment{error=1, leader='evh-pars-dev-centralus.servicebus.windows.net:c:connect-cluster:I:connect-1-f2ebe64c8c3a49b09a7db33b57713f81', leaderUrl='http://10.244.15.4:8083/', offset=669, connectorIds=[], taskIds=[], revokedConnectorIds=[], revokedTaskIds=[], delay=0} with rebalance delay: 0 (org.apache.kafka.connect.runtime.distributed.DistributedHerder) [DistributedHerder-connect-1-1]
2023-06-05 15:06:41,601 INFO [Worker clientId=connect-1, groupId=connect-cluster] Rebalance started (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-06-05 15:06:41,601 INFO [Worker clientId=connect-1, groupId=connect-cluster] (Re-)joining group (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-06-05 15:06:41,692 INFO [Worker clientId=connect-1, groupId=connect-cluster] Successfully joined group with generation Generation{generationId=28916, memberId='evh-pars-dev-centralus.servicebus.windows.net:c:connect-cluster:I:connect-1-805b149901cf40948dfdab5f8f568065', protocol='sessioned'} (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-06-05 15:06:41,694 INFO [Worker clientId=connect-1, groupId=connect-cluster] SyncGroup failed: The group began another rebalance. Need to re-join the group. Sent generation was Generation{generationId=28916, memberId='evh-pars-dev-centralus.servicebus.windows.net:c:connect-cluster:I:connect-1-805b149901cf40948dfdab5f8f568065', protocol='sessioned'} (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-06-05 15:06:41,694 INFO [Worker clientId=connect-1, groupId=connect-cluster] Request joining group due to: rebalance failed due to 'The group is rebalancing, so a rejoin is needed.' (RebalanceInProgressException) (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-06-05 15:06:41,694 INFO [Worker clientId=connect-1, groupId=connect-cluster] (Re-)joining group (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-06-05 15:06:41,710 INFO [Worker clientId=connect-1, groupId=connect-cluster] Successfully joined group with generation Generation{generationId=28917, memberId='evh-pars-dev-centralus.servicebus.windows.net:c:connect-cluster:I:connect-1-805b149901cf40948dfdab5f8f568065', protocol='sessioned'} (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-06-05 15:06:41,726 INFO [Worker clientId=connect-1, groupId=connect-cluster] Successfully synced group in generation Generation{generationId=28917, memberId='evh-pars-dev-centralus.servicebus.windows.net:c:connect-cluster:I:connect-1-805b149901cf40948dfdab5f8f568065', protocol='sessioned'} (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-06-05 15:06:41,726 INFO [Worker clientId=connect-1, groupId=connect-cluster] Joined group at generation 28917 with protocol version 2 and got assignment: Assignment{error=1, leader='evh-pars-dev-centralus.servicebus.windows.net:c:connect-cluster:I:connect-1-f2ebe64c8c3a49b09a7db33b57713f81', leaderUrl='http://10.244.15.4:8083/', offset=669, connectorIds=[], taskIds=[], revokedConnectorIds=[], revokedTaskIds=[], delay=0} with rebalance delay: 0 (org.apache.kafka.connect.runtime.distributed.DistributedHerder) [DistributedHerder-connect-1-1]
2023-06-05 15:06:41,726 INFO [Worker clientId=connect-1, groupId=connect-cluster] Rebalance started (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-06-05 15:06:41,726 INFO [Worker clientId=connect-1, groupId=connect-cluster] (Re-)joining group (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-06-05 15:06:41,849 INFO [Worker clientId=connect-1, groupId=connect-cluster] Successfully joined group with generation Generation{generationId=28918, memberId='evh-pars-dev-centralus.servicebus.windows.net:c:connect-cluster:I:connect-1-805b149901cf40948dfdab5f8f568065', protocol='sessioned'} (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-06-05 15:06:41,866 INFO [Worker clientId=connect-1, groupId=connect-cluster] Successfully synced group in generation Generation{generationId=28918, memberId='evh-pars-dev-centralus.servicebus.windows.net:c:connect-cluster:I:connect-1-805b149901cf40948dfdab5f8f568065', protocol='sessioned'} (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-06-05 15:06:41,866 INFO [Worker clientId=connect-1, groupId=connect-cluster] Joined group at generation 28918 with protocol version 2 and got assignment: Assignment{error=1, leader='evh-pars-dev-centralus.servicebus.windows.net:c:connect-cluster:I:connect-1-f2ebe64c8c3a49b09a7db33b57713f81', leaderUrl='http://10.244.15.4:8083/', offset=669, connectorIds=[], taskIds=[], revokedConnectorIds=[], revokedTaskIds=[], delay=0} with rebalance delay: 0 (org.apache.kafka.connect.runtime.distributed.DistributedHerder) [DistributedHerder-connect-1-1]
2023-06-05 15:06:41,866 INFO [Worker clientId=connect-1, groupId=connect-cluster] Rebalance started (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-06-05 15:06:41,866 INFO [Worker clientId=connect-1, groupId=connect-cluster] (Re-)joining group (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-06-05 15:06:41,882 INFO [Worker clientId=connect-1, groupId=connect-cluster] Successfully joined group with generation Generation{generationId=28919, memberId='evh-pars-dev-centralus.servicebus.windows.net:c:connect-cluster:I:connect-1-805b149901cf40948dfdab5f8f568065', protocol='sessioned'} (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-06-05 15:06:41,897 INFO [Worker clientId=connect-1, groupId=connect-cluster] Successfully synced group in generation Generation{generationId=28919, memberId='evh-pars-dev-centralus.servicebus.windows.net:c:connect-cluster:I:connect-1-805b149901cf40948dfdab5f8f568065', protocol='sessioned'} (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-06-05 15:06:41,898 INFO [Worker clientId=connect-1, groupId=connect-cluster] Joined group at generation 28919 with protocol version 2 and got assignment: Assignment{error=1, leader='evh-pars-dev-centralus.servicebus.windows.net:c:connect-cluster:I:connect-1-f2ebe64c8c3a49b09a7db33b57713f81', leaderUrl='http://10.244.15.4:8083/', offset=669, connectorIds=[], taskIds=[], revokedConnectorIds=[], revokedTaskIds=[], delay=0} with rebalance delay: 0 (org.apache.kafka.connect.runtime.distributed.DistributedHerder) [DistributedHerder-connect-1-1]
2023-06-05 15:06:41,898 INFO [Worker clientId=connect-1, groupId=connect-cluster] Rebalance started (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-06-05 15:06:41,898 INFO [Worker clientId=connect-1, groupId=connect-cluster] (Re-)joining group (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-06-05 15:06:42,020 INFO [Worker clientId=connect-1, groupId=connect-cluster] Successfully joined group with generation Generation{generationId=28920, memberId='evh-pars-dev-centralus.servicebus.windows.net:c:connect-cluster:I:connect-1-805b149901cf40948dfdab5f8f568065', protocol='sessioned'} (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-06-05 15:06:42,038 INFO [Worker clientId=connect-1, groupId=connect-cluster] Successfully synced group in generation Generation{generationId=28920, memberId='evh-pars-dev-centralus.servicebus.windows.net:c:connect-cluster:I:connect-1-805b149901cf40948dfdab5f8f568065', protocol='sessioned'} (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-06-05 15:06:42,038 INFO [Worker clientId=connect-1, groupId=connect-cluster] Joined group at generation 28920 with protocol version 2 and got assignment: Assignment{error=1, leader='evh-pars-dev-centralus.servicebus.windows.net:c:connect-cluster:I:connect-1-f2ebe64c8c3a49b09a7db33b57713f81', leaderUrl='http://10.244.15.4:8083/', offset=669, connectorIds=[], taskIds=[], revokedConnectorIds=[], revokedTaskIds=[], delay=0} with rebalance delay: 0 (org.apache.kafka.connect.runtime.distributed.DistributedHerder) [DistributedHerder-connect-1-1]
2023-06-05 15:06:42,038 INFO [Worker clientId=connect-1, groupId=connect-cluster] Rebalance started (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-06-05 15:06:42,038 INFO [Worker clientId=connect-1, groupId=connect-cluster] (Re-)joining group (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-06-05 15:06:42,145 INFO [Worker clientId=connect-1, groupId=connect-cluster] Successfully joined group with generation Generation{generationId=28920, memberId='evh-pars-dev-centralus.servicebus.windows.net:c:connect-cluster:I:connect-1-805b149901cf40948dfdab5f8f568065', protocol='sessioned'} (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-06-05 15:06:42,163 INFO [Worker clientId=connect-1, groupId=connect-cluster] SyncGroup failed: The group began another rebalance. Need to re-join the group. Sent generation was Generation{generationId=28920, memberId='evh-pars-dev-centralus.servicebus.windows.net:c:connect-cluster:I:connect-1-805b149901cf40948dfdab5f8f568065', protocol='sessioned'} (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-06-05 15:06:42,163 INFO [Worker clientId=connect-1, groupId=connect-cluster] Request joining group due to: rebalance failed due to 'The group is rebalancing, so a rejoin is needed.' (RebalanceInProgressException) (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-06-05 15:06:42,163 INFO [Worker clientId=connect-1, groupId=connect-cluster] (Re-)joining group (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-06-05 15:06:42,286 INFO [Worker clientId=connect-1, groupId=connect-cluster] Successfully joined group with generation Generation{generationId=28921, memberId='evh-pars-dev-centralus.servicebus.windows.net:c:connect-cluster:I:connect-1-805b149901cf40948dfdab5f8f568065', protocol='sessioned'} (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-06-05 15:06:42,319 INFO [Worker clientId=connect-1, groupId=connect-cluster] SyncGroup failed: The group began another rebalance. Need to re-join the group. Sent generation was Generation{generationId=28921, memberId='evh-pars-dev-centralus.servicebus.windows.net:c:connect-cluster:I:connect-1-805b149901cf40948dfdab5f8f568065', protocol='sessioned'} (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-06-05 15:06:42,319 INFO [Worker clientId=connect-1, groupId=connect-cluster] Request joining group due to: rebalance failed due to 'The group is rebalancing, so a rejoin is needed.' (RebalanceInProgressException) (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-06-05 15:06:42,319 INFO [Worker clientId=connect-1, groupId=connect-cluster] (Re-)joining group (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-06-05 15:06:42,335 INFO [Worker clientId=connect-1, groupId=connect-cluster] Successfully joined group with generation Generation{generationId=28922, memberId='evh-pars-dev-centralus.servicebus.windows.net:c:connect-cluster:I:connect-1-805b149901cf40948dfdab5f8f568065', protocol='sessioned'} (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-06-05 15:06:42,350 INFO [Worker clientId=connect-1, groupId=connect-cluster] Successfully synced group in generation Generation{generationId=28922, memberId='evh-pars-dev-centralus.servicebus.windows.net:c:connect-cluster:I:connect-1-805b149901cf40948dfdab5f8f568065', protocol='sessioned'} (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-06-05 15:06:42,350 INFO [Worker clientId=connect-1, groupId=connect-cluster] Joined group at generation 28922 with protocol version 2 and got assignment: Assignment{error=1, leader='evh-pars-dev-centralus.servicebus.windows.net:c:connect-cluster:I:connect-1-f2ebe64c8c3a49b09a7db33b57713f81', leaderUrl='http://10.244.15.4:8083/', offset=669, connectorIds=[], taskIds=[], revokedConnectorIds=[], revokedTaskIds=[], delay=0} with rebalance delay: 0 (org.apache.kafka.connect.runtime.distributed.DistributedHerder) [DistributedHerder-connect-1-1]
2023-06-05 15:06:42,350 INFO [Worker clientId=connect-1, groupId=connect-cluster] Rebalance started (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-06-05 15:06:42,350 INFO [Worker clientId=connect-1, groupId=connect-cluster] (Re-)joining group (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-06-05 15:06:42,457 INFO [Worker clientId=connect-1, groupId=connect-cluster] Successfully joined group with generation Generation{generationId=28922, memberId='evh-pars-dev-centralus.servicebus.windows.net:c:connect-cluster:I:connect-1-805b149901cf40948dfdab5f8f568065', protocol='sessioned'} (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-06-05 15:06:42,460 INFO [Worker clientId=connect-1, groupId=connect-cluster] SyncGroup failed: The group began another rebalance. Need to re-join the group. Sent generation was Generation{generationId=28922, memberId='evh-pars-dev-centralus.servicebus.windows.net:c:connect-cluster:I:connect-1-805b149901cf40948dfdab5f8f568065', protocol='sessioned'} (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-06-05 15:06:42,460 INFO [Worker clientId=connect-1, groupId=connect-cluster] Request joining group due to: rebalance failed due to 'The group is rebalancing, so a rejoin is needed.' (RebalanceInProgressException) (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-06-05 15:06:42,460 INFO [Worker clientId=connect-1, groupId=connect-cluster] (Re-)joining group (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-06-05 15:06:42,476 INFO [Worker clientId=connect-1, groupId=connect-cluster] Successfully joined group with generation Generation{generationId=28923, memberId='evh-pars-dev-centralus.servicebus.windows.net:c:connect-cluster:I:connect-1-805b149901cf40948dfdab5f8f568065', protocol='sessioned'} (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-06-05 15:06:42,491 INFO [Worker clientId=connect-1, groupId=connect-cluster] Successfully synced group in generation Generation{generationId=28923, memberId='evh-pars-dev-centralus.servicebus.windows.net:c:connect-cluster:I:connect-1-805b149901cf40948dfdab5f8f568065', protocol='sessioned'} (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-06-05 15:06:42,491 INFO [Worker clientId=connect-1, groupId=connect-cluster] Joined group at generation 28923 with protocol version 2 and got assignment: Assignment{error=1, leader='evh-pars-dev-centralus.servicebus.windows.net:c:connect-cluster:I:connect-1-f2ebe64c8c3a49b09a7db33b57713f81', leaderUrl='http://10.244.15.4:8083/', offset=669, connectorIds=[], taskIds=[], revokedConnectorIds=[], revokedTaskIds=[], delay=0} with rebalance delay: 0 (org.apache.kafka.connect.runtime.distributed.DistributedHerder) [DistributedHerder-connect-1-1]
2023-06-05 15:06:42,491 INFO [Worker clientId=connect-1, groupId=connect-cluster] Rebalance started (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-06-05 15:06:42,491 INFO [Worker clientId=connect-1, groupId=connect-cluster] (Re-)joining group (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-06-05 15:06:42,598 INFO [Worker clientId=connect-1, groupId=connect-cluster] Successfully joined group with generation Generation{generationId=28924, memberId='evh-pars-dev-centralus.servicebus.windows.net:c:connect-cluster:I:connect-1-805b149901cf40948dfdab5f8f568065', protocol='sessioned'} (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-06-05 15:06:42,616 INFO [Worker clientId=connect-1, groupId=connect-cluster] Successfully synced group in generation Generation{generationId=28924, memberId='evh-pars-dev-centralus.servicebus.windows.net:c:connect-cluster:I:connect-1-805b149901cf40948dfdab5f8f568065', protocol='sessioned'} (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-06-05 15:06:42,616 INFO [Worker clientId=connect-1, groupId=connect-cluster] Joined group at generation 28924 with protocol version 2 and got assignment: Assignment{error=1, leader='evh-pars-dev-centralus.servicebus.windows.net:c:connect-cluster:I:connect-1-f2ebe64c8c3a49b09a7db33b57713f81', leaderUrl='http://10.244.15.4:8083/', offset=669, connectorIds=[], taskIds=[], revokedConnectorIds=[], revokedTaskIds=[], delay=0} with rebalance delay: 0 (org.apache.kafka.connect.runtime.distributed.DistributedHerder) [DistributedHerder-connect-1-1]
2023-06-05 15:06:42,616 INFO [Worker clientId=connect-1, groupId=connect-cluster] Rebalance started (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-06-05 15:06:42,616 INFO [Worker clientId=connect-1, groupId=connect-cluster] (Re-)joining group (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-06-05 15:06:42,739 INFO [Worker clientId=connect-1, groupId=connect-cluster] Successfully joined group with generation Generation{generationId=28925, memberId='evh-pars-dev-centralus.servicebus.windows.net:c:connect-cluster:I:connect-1-805b149901cf40948dfdab5f8f568065', protocol='sessioned'} (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-06-05 15:06:42,757 INFO [Worker clientId=connect-1, groupId=connect-cluster] Successfully synced group in generation Generation{generationId=28925, memberId='evh-pars-dev-centralus.servicebus.windows.net:c:connect-cluster:I:connect-1-805b149901cf40948dfdab5f8f568065', protocol='sessioned'} (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-06-05 15:06:42,757 INFO [Worker clientId=connect-1, groupId=connect-cluster] Joined group at generation 28925 with protocol version 2 and got assignment: Assignment{error=1, leader='evh-pars-dev-centralus.servicebus.windows.net:c:connect-cluster:I:connect-1-f2ebe64c8c3a49b09a7db33b57713f81', leaderUrl='http://10.244.15.4:8083/', offset=669, connectorIds=[], taskIds=[], revokedConnectorIds=[], revokedTaskIds=[], delay=0} with rebalance delay: 0 (org.apache.kafka.connect.runtime.distributed.DistributedHerder) [DistributedHerder-connect-1-1]
2023-06-05 15:06:42,757 INFO [Worker clientId=connect-1, groupId=connect-cluster] Rebalance started (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-06-05 15:06:42,757 INFO [Worker clientId=connect-1, groupId=connect-cluster] (Re-)joining group (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-06-05 15:06:42,864 INFO [Worker clientId=connect-1, groupId=connect-cluster] Successfully joined group with generation Generation{generationId=28926, memberId='evh-pars-dev-centralus.servicebus.windows.net:c:connect-cluster:I:connect-1-805b149901cf40948dfdab5f8f568065', protocol='sessioned'} (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-06-05 15:06:42,890 INFO [Worker clientId=connect-1, groupId=connect-cluster] Successfully synced group in generation Generation{generationId=28926, memberId='evh-pars-dev-centralus.servicebus.windows.net:c:connect-cluster:I:connect-1-805b149901cf40948dfdab5f8f568065', protocol='sessioned'} (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-06-05 15:06:42,890 INFO [Worker clientId=connect-1, groupId=connect-cluster] Joined group at generation 28926 with protocol version 2 and got assignment: Assignment{error=1, leader='evh-pars-dev-centralus.servicebus.windows.net:c:connect-cluster:I:connect-1-f2ebe64c8c3a49b09a7db33b57713f81', leaderUrl='http://10.244.15.4:8083/', offset=669, connectorIds=[], taskIds=[], revokedConnectorIds=[], revokedTaskIds=[], delay=0} with rebalance delay: 0 (org.apache.kafka.connect.runtime.distributed.DistributedHerder) [DistributedHerder-connect-1-1]
2023-06-05 15:06:42,890 INFO [Worker clientId=connect-1, groupId=connect-cluster] Rebalance started (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-06-05 15:06:42,890 INFO [Worker clientId=connect-1, groupId=connect-cluster] (Re-)joining group (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-06-05 15:06:42,989 INFO [Worker clientId=connect-1, groupId=connect-cluster] Successfully joined group with generation Generation{generationId=28927, memberId='evh-pars-dev-centralus.servicebus.windows.net:c:connect-cluster:I:connect-1-805b149901cf40948dfdab5f8f568065', protocol='sessioned'} (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-06-05 15:06:43,007 INFO [Worker clientId=connect-1, groupId=connect-cluster] Successfully synced group in generation Generation{generationId=28927, memberId='evh-pars-dev-centralus.servicebus.windows.net:c:connect-cluster:I:connect-1-805b149901cf40948dfdab5f8f568065', protocol='sessioned'} (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-06-05 15:06:43,007 INFO [Worker clientId=connect-1, groupId=connect-cluster] Joined group at generation 28927 with protocol version 2 and got assignment: Assignment{error=1, leader='evh-pars-dev-centralus.servicebus.windows.net:c:connect-cluster:I:connect-1-f2ebe64c8c3a49b09a7db33b57713f81', leaderUrl='http://10.244.15.4:8083/', offset=669, connectorIds=[], taskIds=[], revokedConnectorIds=[], revokedTaskIds=[], delay=0} with rebalance delay: 0 (org.apache.kafka.connect.runtime.distributed.DistributedHerder) [DistributedHerder-connect-1-1]
2023-06-05 15:06:43,007 INFO [Worker clientId=connect-1, groupId=connect-cluster] Rebalance started (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-06-05 15:06:43,007 INFO [Worker clientId=connect-1, groupId=connect-cluster] (Re-)joining group (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-06-05 15:06:43,129 INFO [Worker clientId=connect-1, groupId=connect-cluster] Successfully joined group with generation Generation{generationId=28928, memberId='evh-pars-dev-centralus.servicebus.windows.net:c:connect-cluster:I:connect-1-805b149901cf40948dfdab5f8f568065', protocol='sessioned'} (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-06-05 15:06:43,147 INFO [Worker clientId=connect-1, groupId=connect-cluster] Successfully synced group in generation Generation{generationId=28928, memberId='evh-pars-dev-centralus.servicebus.windows.net:c:connect-cluster:I:connect-1-805b149901cf40948dfdab5f8f568065', protocol='sessioned'} (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-06-05 15:06:43,148 INFO [Worker clientId=connect-1, groupId=connect-cluster] Joined group at generation 28928 with protocol version 2 and got assignment: Assignment{error=1, leader='evh-pars-dev-centralus.servicebus.windows.net:c:connect-cluster:I:connect-1-f2ebe64c8c3a49b09a7db33b57713f81', leaderUrl='http://10.244.15.4:8083/', offset=669, connectorIds=[], taskIds=[], revokedConnectorIds=[], revokedTaskIds=[], delay=0} with rebalance delay: 0 (org.apache.kafka.connect.runtime.distributed.DistributedHerder) [DistributedHerder-connect-1-1]
2023-06-05 15:06:43,148 INFO [Worker clientId=connect-1, groupId=connect-cluster] Rebalance started (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-06-05 15:06:43,148 INFO [Worker clientId=connect-1, groupId=connect-cluster] (Re-)joining group (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-06-05 15:06:43,163 INFO [Worker clientId=connect-1, groupId=connect-cluster] Successfully joined group with generation Generation{generationId=28929, memberId='evh-pars-dev-centralus.servicebus.windows.net:c:connect-cluster:I:connect-1-805b149901cf40948dfdab5f8f568065', protocol='sessioned'} (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-06-05 15:06:43,178 INFO [Worker clientId=connect-1, groupId=connect-cluster] Successfully synced group in generation Generation{generationId=28929, memberId='evh-pars-dev-centralus.servicebus.windows.net:c:connect-cluster:I:connect-1-805b149901cf40948dfdab5f8f568065', protocol='sessioned'} (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-06-05 15:06:43,179 INFO [Worker clientId=connect-1, groupId=connect-cluster] Joined group at generation 28929 with protocol version 2 and got assignment: Assignment{error=1, leader='evh-pars-dev-centralus.servicebus.windows.net:c:connect-cluster:I:connect-1-f2ebe64c8c3a49b09a7db33b57713f81', leaderUrl='http://10.244.15.4:8083/', offset=669, connectorIds=[], taskIds=[], revokedConnectorIds=[], revokedTaskIds=[], delay=0} with rebalance delay: 0 (org.apache.kafka.connect.runtime.distributed.DistributedHerder) [DistributedHerder-connect-1-1]
2023-06-05 15:06:43,179 INFO [Worker clientId=connect-1, groupId=connect-cluster] Rebalance started (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-06-05 15:06:43,179 INFO [Worker clientId=connect-1, groupId=connect-cluster] (Re-)joining group (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-06-05 15:06:43,301 INFO [Worker clientId=connect-1, groupId=connect-cluster] Successfully joined group with generation Generation{generationId=28930, memberId='evh-pars-dev-centralus.servicebus.windows.net:c:connect-cluster:I:connect-1-805b149901cf40948dfdab5f8f568065', protocol='sessioned'} (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-06-05 15:06:43,339 INFO [Worker clientId=connect-1, groupId=connect-cluster] Successfully synced group in generation Generation{generationId=28930, memberId='evh-pars-dev-centralus.servicebus.windows.net:c:connect-cluster:I:connect-1-805b149901cf40948dfdab5f8f568065', protocol='sessioned'} (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [kafka-coordinator-heartbeat-thread | connect-cluster]
2023-06-05 15:06:43,339 INFO [Worker clientId=connect-1, groupId=connect-cluster] Joined group at generation 28930 with protocol version 2 and got assignment: Assignment{error=1, leader='evh-pars-dev-centralus.servicebus.windows.net:c:connect-cluster:I:connect-1-f2ebe64c8c3a49b09a7db33b57713f81', leaderUrl='http://10.244.15.4:8083/', offset=669, connectorIds=[], taskIds=[], revokedConnectorIds=[], revokedTaskIds=[], delay=0} with rebalance delay: 0 (org.apache.kafka.connect.runtime.distributed.DistributedHerder) [DistributedHerder-connect-1-1]
2023-06-05 15:06:43,339 INFO [Worker clientId=connect-1, groupId=connect-cluster] Rebalance started (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-06-05 15:06:43,339 INFO [Worker clientId=connect-1, groupId=connect-cluster] (Re-)joining group (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-06-05 15:06:43,351 INFO [Worker clientId=connect-1, groupId=connect-cluster] Successfully joined group with generation Generation{generationId=28931, memberId='evh-pars-dev-centralus.servicebus.windows.net:c:connect-cluster:I:connect-1-805b149901cf40948dfdab5f8f568065', protocol='sessioned'} (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-06-05 15:06:43,366 INFO [Worker clientId=connect-1, groupId=connect-cluster] Successfully synced group in generation Generation{generationId=28931, memberId='evh-pars-dev-centralus.servicebus.windows.net:c:connect-cluster:I:connect-1-805b149901cf40948dfdab5f8f568065', protocol='sessioned'} (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-06-05 15:06:43,366 INFO [Worker clientId=connect-1, groupId=connect-cluster] Joined group at generation 28931 with protocol version 2 and got assignment: Assignment{error=1, leader='evh-pars-dev-centralus.servicebus.windows.net:c:connect-cluster:I:connect-1-f2ebe64c8c3a49b09a7db33b57713f81', leaderUrl='http://10.244.15.4:8083/', offset=669, connectorIds=[], taskIds=[], revokedConnectorIds=[], revokedTaskIds=[], delay=0} with rebalance delay: 0 (org.apache.kafka.connect.runtime.distributed.DistributedHerder) [DistributedHerder-connect-1-1]
2023-06-05 15:06:43,366 INFO [Worker clientId=connect-1, groupId=connect-cluster] Rebalance started (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-06-05 15:06:43,366 INFO [Worker clientId=connect-1, groupId=connect-cluster] (Re-)joining group (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-06-05 15:06:43,489 INFO [Worker clientId=connect-1, groupId=connect-cluster] Successfully joined group with generation Generation{generationId=28932, memberId='evh-pars-dev-centralus.servicebus.windows.net:c:connect-cluster:I:connect-1-805b149901cf40948dfdab5f8f568065', protocol='sessioned'} (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-06-05 15:06:43,539 INFO [Worker clientId=connect-1, groupId=connect-cluster] Successfully synced group in generation Generation{generationId=28932, memberId='evh-pars-dev-centralus.servicebus.windows.net:c:connect-cluster:I:connect-1-805b149901cf40948dfdab5f8f568065', protocol='sessioned'} (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-06-05 15:06:43,539 INFO [Worker clientId=connect-1, groupId=connect-cluster] Joined group at generation 28932 with protocol version 2 and got assignment: Assignment{error=1, leader='evh-pars-dev-centralus.servicebus.windows.net:c:connect-cluster:I:connect-1-f2ebe64c8c3a49b09a7db33b57713f81', leaderUrl='http://10.244.15.4:8083/', offset=669, connectorIds=[], taskIds=[], revokedConnectorIds=[], revokedTaskIds=[], delay=0} with rebalance delay: 0 (org.apache.kafka.connect.runtime.distributed.DistributedHerder) [DistributedHerder-connect-1-1]
2023-06-05 15:06:43,539 INFO [Worker clientId=connect-1, groupId=connect-cluster] Rebalance started (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-06-05 15:06:43,539 INFO [Worker clientId=connect-1, groupId=connect-cluster] (Re-)joining group (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-06-05 15:06:43,629 INFO [Worker clientId=connect-1, groupId=connect-cluster] Successfully joined group with generation Generation{generationId=28933, memberId='evh-pars-dev-centralus.servicebus.windows.net:c:connect-cluster:I:connect-1-805b149901cf40948dfdab5f8f568065', protocol='sessioned'} (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-06-05 15:06:43,647 INFO [Worker clientId=connect-1, groupId=connect-cluster] Successfully synced group in generation Generation{generationId=28933, memberId='evh-pars-dev-centralus.servicebus.windows.net:c:connect-cluster:I:connect-1-805b149901cf40948dfdab5f8f568065', protocol='sessioned'} (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-06-05 15:06:43,647 INFO [Worker clientId=connect-1, groupId=connect-cluster] Joined group at generation 28933 with protocol version 2 and got assignment: Assignment{error=1, leader='evh-pars-dev-centralus.servicebus.windows.net:c:connect-cluster:I:connect-1-f2ebe64c8c3a49b09a7db33b57713f81', leaderUrl='http://10.244.15.4:8083/', offset=669, connectorIds=[], taskIds=[], revokedConnectorIds=[], revokedTaskIds=[], delay=0} with rebalance delay: 0 (org.apache.kafka.connect.runtime.distributed.DistributedHerder) [DistributedHerder-connect-1-1]
2023-06-05 15:06:43,648 INFO [Worker clientId=connect-1, groupId=connect-cluster] Rebalance started (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-06-05 15:06:43,648 INFO [Worker clientId=connect-1, groupId=connect-cluster] (Re-)joining group (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-06-05 15:06:43,770 INFO [Worker clientId=connect-1, groupId=connect-cluster] Successfully joined group with generation Generation{generationId=28934, memberId='evh-pars-dev-centralus.servicebus.windows.net:c:connect-cluster:I:connect-1-805b149901cf40948dfdab5f8f568065', protocol='sessioned'} (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-06-05 15:06:43,788 INFO [Worker clientId=connect-1, groupId=connect-cluster] Successfully synced group in generation Generation{generationId=28934, memberId='evh-pars-dev-centralus.servicebus.windows.net:c:connect-cluster:I:connect-1-805b149901cf40948dfdab5f8f568065', protocol='sessioned'} (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-06-05 15:06:43,788 INFO [Worker clientId=connect-1, groupId=connect-cluster] Joined group at generation 28934 with protocol version 2 and got assignment: Assignment{error=1, leader='evh-pars-dev-centralus.servicebus.windows.net:c:connect-cluster:I:connect-1-f2ebe64c8c3a49b09a7db33b57713f81', leaderUrl='http://10.244.15.4:8083/', offset=669, connectorIds=[], taskIds=[], revokedConnectorIds=[], revokedTaskIds=[], delay=0} with rebalance delay: 0 (org.apache.kafka.connect.runtime.distributed.DistributedHerder) [DistributedHerder-connect-1-1]
2023-06-05 15:06:43,788 INFO [Worker clientId=connect-1, groupId=connect-cluster] Rebalance started (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-06-05 15:06:43,788 INFO [Worker clientId=connect-1, groupId=connect-cluster] (Re-)joining group (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-06-05 15:06:43,820 INFO [Worker clientId=connect-1, groupId=connect-cluster] Successfully joined group with generation Generation{generationId=28935, memberId='evh-pars-dev-centralus.servicebus.windows.net:c:connect-cluster:I:connect-1-805b149901cf40948dfdab5f8f568065', protocol='sessioned'} (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-06-05 15:06:43,835 INFO [Worker clientId=connect-1, groupId=connect-cluster] Successfully synced group in generation Generation{generationId=28935, memberId='evh-pars-dev-centralus.servicebus.windows.net:c:connect-cluster:I:connect-1-805b149901cf40948dfdab5f8f568065', protocol='sessioned'} (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-06-05 15:06:43,835 INFO [Worker clientId=connect-1, groupId=connect-cluster] Joined group at generation 28935 with protocol version 2 and got assignment: Assignment{error=1, leader='evh-pars-dev-centralus.servicebus.windows.net:c:connect-cluster:I:connect-1-f2ebe64c8c3a49b09a7db33b57713f81', leaderUrl='http://10.244.15.4:8083/', offset=669, connectorIds=[], taskIds=[], revokedConnectorIds=[], revokedTaskIds=[], delay=0} with rebalance delay: 0 (org.apache.kafka.connect.runtime.distributed.DistributedHerder) [DistributedHerder-connect-1-1]
2023-06-05 15:06:43,835 INFO [Worker clientId=connect-1, groupId=connect-cluster] Rebalance started (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-06-05 15:06:43,835 INFO [Worker clientId=connect-1, groupId=connect-cluster] (Re-)joining group (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-06-05 15:06:43,851 INFO [Worker clientId=connect-1, groupId=connect-cluster] Successfully joined group with generation Generation{generationId=28936, memberId='evh-pars-dev-centralus.servicebus.windows.net:c:connect-cluster:I:connect-1-805b149901cf40948dfdab5f8f568065', protocol='sessioned'} (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-06-05 15:06:43,866 INFO [Worker clientId=connect-1, groupId=connect-cluster] Successfully synced group in generation Generation{generationId=28936, memberId='evh-pars-dev-centralus.servicebus.windows.net:c:connect-cluster:I:connect-1-805b149901cf40948dfdab5f8f568065', protocol='sessioned'} (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-06-05 15:06:43,866 INFO [Worker clientId=connect-1, groupId=connect-cluster] Joined group at generation 28936 with protocol version 2 and got assignment: Assignment{error=1, leader='evh-pars-dev-centralus.servicebus.windows.net:c:connect-cluster:I:connect-1-f2ebe64c8c3a49b09a7db33b57713f81', leaderUrl='http://10.244.15.4:8083/', offset=669, connectorIds=[], taskIds=[], revokedConnectorIds=[], revokedTaskIds=[], delay=0} with rebalance delay: 0 (org.apache.kafka.connect.runtime.distributed.DistributedHerder) [DistributedHerder-connect-1-1]
2023-06-05 15:06:43,866 INFO [Worker clientId=connect-1, groupId=connect-cluster] Rebalance started (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-06-05 15:06:43,866 INFO [Worker clientId=connect-1, groupId=connect-cluster] (Re-)joining group (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-06-05 15:06:43,973 INFO [Worker clientId=connect-1, groupId=connect-cluster] Successfully joined group with generation Generation{generationId=28936, memberId='evh-pars-dev-centralus.servicebus.windows.net:c:connect-cluster:I:connect-1-805b149901cf40948dfdab5f8f568065', protocol='sessioned'} (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-06-05 15:06:43,991 INFO [Worker clientId=connect-1, groupId=connect-cluster] SyncGroup failed: The group began another rebalance. Need to re-join the group. Sent generation was Generation{generationId=28936, memberId='evh-pars-dev-centralus.servicebus.windows.net:c:connect-cluster:I:connect-1-805b149901cf40948dfdab5f8f568065', protocol='sessioned'} (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-06-05 15:06:43,991 INFO [Worker clientId=connect-1, groupId=connect-cluster] Request joining group due to: rebalance failed due to 'The group is rebalancing, so a rejoin is needed.' (RebalanceInProgressException) (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-06-05 15:06:43,991 INFO [Worker clientId=connect-1, groupId=connect-cluster] (Re-)joining group (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-06-05 15:06:44,098 INFO [Worker clientId=connect-1, groupId=connect-cluster] Successfully joined group with generation Generation{generationId=28937, memberId='evh-pars-dev-centralus.servicebus.windows.net:c:connect-cluster:I:connect-1-805b149901cf40948dfdab5f8f568065', protocol='sessioned'} (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-06-05 15:06:44,116 INFO [Worker clientId=connect-1, groupId=connect-cluster] Successfully synced group in generation Generation{generationId=28937, memberId='evh-pars-dev-centralus.servicebus.windows.net:c:connect-cluster:I:connect-1-805b149901cf40948dfdab5f8f568065', protocol='sessioned'} (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-06-05 15:06:44,116 INFO [Worker clientId=connect-1, groupId=connect-cluster] Joined group at generation 28937 with protocol version 2 and got assignment: Assignment{error=1, leader='evh-pars-dev-centralus.servicebus.windows.net:c:connect-cluster:I:connect-1-f2ebe64c8c3a49b09a7db33b57713f81', leaderUrl='http://10.244.15.4:8083/', offset=669, connectorIds=[], taskIds=[], revokedConnectorIds=[], revokedTaskIds=[], delay=0} with rebalance delay: 0 (org.apache.kafka.connect.runtime.distributed.DistributedHerder) [DistributedHerder-connect-1-1]
2023-06-05 15:06:44,116 INFO [Worker clientId=connect-1, groupId=connect-cluster] Rebalance started (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-06-05 15:06:44,116 INFO [Worker clientId=connect-1, groupId=connect-cluster] (Re-)joining group (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-06-05 15:06:44,132 INFO [Worker clientId=connect-1, groupId=connect-cluster] Successfully joined group with generation Generation{generationId=28938, memberId='evh-pars-dev-centralus.servicebus.windows.net:c:connect-cluster:I:connect-1-805b149901cf40948dfdab5f8f568065', protocol='sessioned'} (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-06-05 15:06:44,147 INFO [Worker clientId=connect-1, groupId=connect-cluster] Successfully synced group in generation Generation{generationId=28938, memberId='evh-pars-dev-centralus.servicebus.windows.net:c:connect-cluster:I:connect-1-805b149901cf40948dfdab5f8f568065', protocol='sessioned'} (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-06-05 15:06:44,148 INFO [Worker clientId=connect-1, groupId=connect-cluster] Joined group at generation 28938 with protocol version 2 and got assignment: Assignment{error=1, leader='evh-pars-dev-centralus.servicebus.windows.net:c:connect-cluster:I:connect-1-f2ebe64c8c3a49b09a7db33b57713f81', leaderUrl='http://10.244.15.4:8083/', offset=669, connectorIds=[], taskIds=[], revokedConnectorIds=[], revokedTaskIds=[], delay=0} with rebalance delay: 0 (org.apache.kafka.connect.runtime.distributed.DistributedHerder) [DistributedHerder-connect-1-1]
2023-06-05 15:06:44,148 INFO [Worker clientId=connect-1, groupId=connect-cluster] Rebalance started (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-06-05 15:06:44,148 INFO [Worker clientId=connect-1, groupId=connect-cluster] (Re-)joining group (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-06-05 15:06:44,254 ERROR [Worker clientId=connect-1, groupId=connect-cluster] JoinGroup failed due to unexpected error: The server experienced an unexpected error when processing the request. (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-06-05 15:06:44,255 INFO [Worker clientId=connect-1, groupId=connect-cluster] Request joining group due to: rebalance failed due to 'Unexpected error in join group response: The server experienced an unexpected error when processing the request.' (KafkaException) (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator) [DistributedHerder-connect-1-1]
2023-06-05 15:06:44,255 ERROR [Worker clientId=connect-1, groupId=connect-cluster] Uncaught exception in herder work thread, exiting:  (org.apache.kafka.connect.runtime.distributed.DistributedHerder) [DistributedHerder-connect-1-1]
org.apache.kafka.common.KafkaException: Unexpected error in join group response: The server experienced an unexpected error when processing the request.
        at org.apache.kafka.clients.consumer.internals.AbstractCoordinator$JoinGroupResponseHandler.handle(AbstractCoordinator.java:701)
        at org.apache.kafka.clients.consumer.internals.AbstractCoordinator$JoinGroupResponseHandler.handle(AbstractCoordinator.java:593)
        at org.apache.kafka.clients.consumer.internals.AbstractCoordinator$CoordinatorResponseHandler.onSuccess(AbstractCoordinator.java:1260)
        at org.apache.kafka.clients.consumer.internals.AbstractCoordinator$CoordinatorResponseHandler.onSuccess(AbstractCoordinator.java:1235)
        at org.apache.kafka.clients.consumer.internals.RequestFuture$1.onSuccess(RequestFuture.java:206)
        at org.apache.kafka.clients.consumer.internals.RequestFuture.fireSuccess(RequestFuture.java:169)
        at org.apache.kafka.clients.consumer.internals.RequestFuture.complete(RequestFuture.java:129)
        at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient$RequestFutureCompletionHandler.fireCompletion(ConsumerNetworkClient.java:617)
        at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.firePendingCompletedRequests(ConsumerNetworkClient.java:427)
        at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:312)
        at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:230)
        at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:214)
        at org.apache.kafka.clients.consumer.internals.AbstractCoordinator.joinGroupIfNeeded(AbstractCoordinator.java:451)
        at org.apache.kafka.clients.consumer.internals.AbstractCoordinator.ensureActiveGroup(AbstractCoordinator.java:385)
        at org.apache.kafka.clients.consumer.internals.AbstractCoordinator.ensureActiveGroup(AbstractCoordinator.java:365)
        at org.apache.kafka.connect.runtime.distributed.WorkerCoordinator.poll(WorkerCoordinator.java:149)
        at org.apache.kafka.connect.runtime.distributed.WorkerGroupMember.ensureActive(WorkerGroupMember.java:161)
        at org.apache.kafka.connect.runtime.distributed.DistributedHerder.tick(DistributedHerder.java:386)
        at org.apache.kafka.connect.runtime.distributed.DistributedHerder.run(DistributedHerder.java:352)
        at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
        at java.base/java.lang.Thread.run(Thread.java:833)
2023-06-05 15:06:44,258 INFO Kafka Connect stopping (org.apache.kafka.connect.runtime.Connect) [connect-shutdown-hook]
2023-06-05 15:06:44,258 INFO Stopping REST server (org.apache.kafka.connect.runtime.rest.RestServer) [connect-shutdown-hook]
2023-06-05 15:06:44,260 INFO Stopped http_8083@b8142f4{HTTP/1.1, (http/1.1)}{0.0.0.0:8083} (org.eclipse.jetty.server.AbstractConnector) [JettyShutdownThread]
2023-06-05 15:06:44,260 INFO node0 Stopped scavenging (org.eclipse.jetty.server.session) [JettyShutdownThread]
2023-06-05 15:06:44,278 INFO REST server stopped (org.apache.kafka.connect.runtime.rest.RestServer) [connect-shutdown-hook]
2023-06-05 15:06:44,278 INFO [Worker clientId=connect-1, groupId=connect-cluster] Herder stopping (org.apache.kafka.connect.runtime.distributed.DistributedHerder) [connect-shutdown-hook]
2023-06-05 15:06:49,279 INFO [Worker clientId=connect-1, groupId=connect-cluster] Herder stopped (org.apache.kafka.connect.runtime.distributed.DistributedHerder) [connect-shutdown-hook]
2023-06-05 15:06:49,279 INFO Kafka Connect stopped (org.apache.kafka.connect.runtime.Connect) [connect-shutdown-hook]

imrohankataria Jun 5, 2023
Author

Current state of pod

pars-kc-connect-7864579d79-xlgnr            0/1     CrashLoopBackOff         29 (51s ago)   105m

Current Events

LAST SEEN   TYPE      REASON              OBJECT                                  MESSAGE
15m         Normal    Scheduled           pod/debezium-ui-7dc4b5cb84-pmcz4        Successfully assigned kafka-connect/debezium-ui-7dc4b5cb84-pmcz4 to aks-agentpool-38288350-vmss00000c
15m         Normal    Pulling             pod/debezium-ui-7dc4b5cb84-pmcz4        Pulling image "debezium/debezium-ui:latest"
15m         Normal    Pulled              pod/debezium-ui-7dc4b5cb84-pmcz4        Successfully pulled image "debezium/debezium-ui:latest" in 6.309732801s
15m         Normal    Created             pod/debezium-ui-7dc4b5cb84-pmcz4        Created container debezium-ui
15m         Normal    Started             pod/debezium-ui-7dc4b5cb84-pmcz4        Started container debezium-ui
15m         Normal    Killing             pod/debezium-ui-7dc4b5cb84-pmcz4        Stopping container debezium-ui
15m         Normal    SuccessfulCreate    replicaset/debezium-ui-7dc4b5cb84       Created pod: debezium-ui-7dc4b5cb84-pmcz4
15m         Normal    SuccessfulDelete    replicaset/debezium-ui-7dc4b5cb84       Deleted pod: debezium-ui-7dc4b5cb84-pmcz4
15m         Normal    ScalingReplicaSet   deployment/debezium-ui                  Scaled up replica set debezium-ui-7dc4b5cb84 to 1
15m         Normal    ScalingReplicaSet   deployment/debezium-ui                  Scaled down replica set debezium-ui-7dc4b5cb84 to 0
21m         Normal    Created             pod/pars-kc-connect-6cdb86d649-28c2r    Created container pars-kc-connect
21m         Normal    Started             pod/pars-kc-connect-6cdb86d649-28c2r    Started container pars-kc-connect
21m         Warning   Unhealthy           pod/pars-kc-connect-6cdb86d649-28c2r    Readiness probe failed: Get "http://10.244.15.6:8083/": dial tcp 10.244.15.6:8083: connect: connection refused
21m         Warning   Unhealthy           pod/pars-kc-connect-6cdb86d649-28c2r    Liveness probe failed: Get "http://10.244.15.6:8083/": dial tcp 10.244.15.6:8083: connect: connection refused
21m         Normal    Pulled              pod/pars-kc-connect-6cdb86d649-28c2r    Container image "acrparsdevcentralus.azurecr.io/connect@sha256:843dbf5b64340b48bb0388d3b54c4d071fb10fbe8d3bf96cbfd4afe3210f064c" already present on machine
17m         Warning   BackOff             pod/pars-kc-connect-6cdb86d649-28c2r    Back-off restarting failed container
4m2s        Normal    Killing             pod/pars-kc-connect-6cdb86d649-28c2r    Stopping container pars-kc-connect
4m2s        Normal    SuccessfulDelete    replicaset/pars-kc-connect-6cdb86d649   Deleted pod: pars-kc-connect-6cdb86d649-28c2r
11m         Warning   Unhealthy           pod/pars-kc-connect-7864579d79-xlgnr    Liveness probe failed: Get "http://10.244.0.33:8083/": dial tcp 10.244.0.33:8083: connect: connection refused
21m         Warning   Unhealthy           pod/pars-kc-connect-7864579d79-xlgnr    Readiness probe failed: Get "http://10.244.0.33:8083/": dial tcp 10.244.0.33:8083: connect: connection refused
46m         Normal    Pulled              pod/pars-kc-connect-7864579d79-xlgnr    Container image "acrparsdevcentralus.azurecr.io/connect@sha256:843dbf5b64340b48bb0388d3b54c4d071fb10fbe8d3bf96cbfd4afe3210f064c" already present on machine
105s        Warning   BackOff             pod/pars-kc-connect-7864579d79-xlgnr    Back-off restarting failed container
4m2s        Normal    ScalingReplicaSet   deployment/pars-kc-connect              Scaled down replica set pars-kc-connect-6cdb86d649 to 0

Operator logs

2023-06-05 15:15:49 ERROR Util:166 - Reconciliation #1912(timer) KafkaConnect(kafka-connect/pars-kc): Exceeded timeout of 300000ms while waiting for Deployment resource pars-kc-connect in namespace kafka-connect to be ready
2023-06-05 15:15:49 ERROR AbstractOperator:260 - Reconciliation #1912(timer) KafkaConnect(kafka-connect/pars-kc): createOrUpdate failed
io.strimzi.operator.common.operator.resource.TimeoutException: Exceeded timeout of 300000ms while waiting for Deployment resource pars-kc-connect in namespace kafka-connect to be ready
        at io.strimzi.operator.common.Util$1.lambda$handle$1(Util.java:167) ~[io.strimzi.operator-common-0.35.0.jar:0.35.0]
        at io.vertx.core.impl.future.FutureImpl$3.onFailure(FutureImpl.java:153) ~[io.vertx.vertx-core-4.3.8.jar:4.3.8]
        at io.vertx.core.impl.future.FutureBase.lambda$emitFailure$1(FutureBase.java:69) ~[io.vertx.vertx-core-4.3.8.jar:4.3.8]
        at io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:174) ~[io.netty.netty-common-4.1.87.Final.jar:4.1.87.Final]
        at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:167) ~[io.netty.netty-common-4.1.87.Final.jar:4.1.87.Final]
        at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:470) ~[io.netty.netty-common-4.1.87.Final.jar:4.1.87.Final]
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:569) ~[io.netty.netty-transport-4.1.87.Final.jar:4.1.87.Final]
        at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) ~[io.netty.netty-common-4.1.87.Final.jar:4.1.87.Final]
        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[io.netty.netty-common-4.1.87.Final.jar:4.1.87.Final]
        at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[io.netty.netty-common-4.1.87.Final.jar:4.1.87.Final]
        at java.lang.Thread.run(Thread.java:833) ~[?:?]
2023-06-05 15:15:49 WARN  AbstractOperator:525 - Reconciliation #1912(timer) KafkaConnect(kafka-connect/pars-kc): Failed to reconcile
io.strimzi.operator.common.operator.resource.TimeoutException: Exceeded timeout of 300000ms while waiting for Deployment resource pars-kc-connect in namespace kafka-connect to be ready
        at io.strimzi.operator.common.Util$1.lambda$handle$1(Util.java:167) ~[io.strimzi.operator-common-0.35.0.jar:0.35.0]
        at io.vertx.core.impl.future.FutureImpl$3.onFailure(FutureImpl.java:153) ~[io.vertx.vertx-core-4.3.8.jar:4.3.8]
        at io.vertx.core.impl.future.FutureBase.lambda$emitFailure$1(FutureBase.java:69) ~[io.vertx.vertx-core-4.3.8.jar:4.3.8]
        at io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:174) ~[io.netty.netty-common-4.1.87.Final.jar:4.1.87.Final]
        at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:167) ~[io.netty.netty-common-4.1.87.Final.jar:4.1.87.Final]
        at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:470) ~[io.netty.netty-common-4.1.87.Final.jar:4.1.87.Final]
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:569) ~[io.netty.netty-transport-4.1.87.Final.jar:4.1.87.Final]
        at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) ~[io.netty.netty-common-4.1.87.Final.jar:4.1.87.Final]
        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[io.netty.netty-common-4.1.87.Final.jar:4.1.87.Final]
        at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[io.netty.netty-common-4.1.87.Final.jar:4.1.87.Final]
        at java.lang.Thread.run(Thread.java:833) ~[?:?]
2023-06-05 15:16:48 INFO  ClusterOperator:139 - Triggering periodic reconciliation for namespace kafka-connect
2023-06-05 15:16:48 WARN  StatusUtils:151 - Reconciliation #1918(timer) KafkaConnect(kafka-connect/pars-kc): Contains object at path spec.build.plugins.artifacts with an unknown property: version
2023-06-05 15:16:48 WARN  StatusUtils:151 - Reconciliation #1918(timer) KafkaConnect(kafka-connect/pars-kc): Contains object at path spec.build.plugins.artifacts with an unknown property: version
2023-06-05 15:16:48 INFO  AbstractOperator:239 - Reconciliation #1918(timer) KafkaConnect(kafka-connect/pars-kc): KafkaConnect pars-kc will be checked for creation or modification
2023-06-05 15:16:48 WARN  StatusUtils:151 - Reconciliation #1919(timer) KafkaConnect(kafka-connect/ccat): Contains object at path spec.build.plugins.artifacts with an unknown property: version
2023-06-05 15:16:48 WARN  StatusUtils:151 - Reconciliation #1919(timer) KafkaConnect(kafka-connect/ccat): Contains object at path spec.build.plugins.artifacts with an unknown property: version
2023-06-05 15:16:48 INFO  AbstractOperator:239 - Reconciliation #1919(timer) KafkaConnect(kafka-connect/ccat): KafkaConnect ccat will be checked for creation or modification
2023-06-05 15:16:49 INFO  AbstractConnectOperator:502 - Reconciliation #1919(timer) KafkaConnect(kafka-connect/ccat): creating/updating connector: ccat-connector-source-bulk
2023-06-05 15:16:49 INFO  AbstractConnectOperator:502 - Reconciliation #1919(timer) KafkaConnect(kafka-connect/ccat): creating/updating connector: ccat-sink-connector
2023-06-05 15:17:35 INFO  AbstractOperator:510 - Reconciliation #1919(timer) KafkaConnect(kafka-connect/ccat): reconciled
2023-06-05 15:17:41 INFO  AbstractConnectOperator:502 - Reconciliation #1918(timer) KafkaConnect(kafka-connect/pars-kc): creating/updating connector: ccat-connector-sink
2023-06-05 15:17:41 INFO  AbstractConnectOperator:502 - Reconciliation #1918(timer) KafkaConnect(kafka-connect/pars-kc): creating/updating connector: artemis-connector
2023-06-05 15:17:41 INFO  AbstractConnectOperator:502 - Reconciliation #1918(timer) KafkaConnect(kafka-connect/pars-kc): creating/updating connector: artemis-connector-sink
2023-06-05 15:17:41 INFO  AbstractConnectOperator:502 - Reconciliation #1918(timer) KafkaConnect(kafka-connect/pars-kc): creating/updating connector: ccat-connector-source-incr
2023-06-05 15:17:48 INFO  AbstractOperator:380 - Reconciliation #1918(timer) KafkaConnect(kafka-connect/pars-kc): Reconciliation is in progress
2023-06-05 15:18:48 INFO  ClusterOperator:139 - Triggering periodic reconciliation for namespace kafka-connect
2023-06-05 15:18:48 WARN  StatusUtils:151 - Reconciliation #1921(timer) KafkaConnect(kafka-connect/ccat): Contains object at path spec.build.plugins.artifacts with an unknown property: version
2023-06-05 15:18:48 WARN  StatusUtils:151 - Reconciliation #1921(timer) KafkaConnect(kafka-connect/ccat): Contains object at path spec.build.plugins.artifacts with an unknown property: version
2023-06-05 15:18:48 INFO  AbstractOperator:239 - Reconciliation #1921(timer) KafkaConnect(kafka-connect/ccat): KafkaConnect ccat will be checked for creation or modification
2023-06-05 15:18:48 INFO  AbstractOperator:380 - Reconciliation #1918(timer) KafkaConnect(kafka-connect/pars-kc): Reconciliation is in progress
2023-06-05 15:18:49 INFO  AbstractConnectOperator:502 - Reconciliation #1921(timer) KafkaConnect(kafka-connect/ccat): creating/updating connector: ccat-sink-connector
2023-06-05 15:18:49 INFO  AbstractConnectOperator:502 - Reconciliation #1921(timer) KafkaConnect(kafka-connect/ccat): creating/updating connector: ccat-connector-source-bulk
2023-06-05 15:18:49 INFO  AbstractOperator:510 - Reconciliation #1921(timer) KafkaConnect(kafka-connect/ccat): reconciled
2023-06-05 15:19:11 WARN  AbstractConnectOperator:944 - Reconciliation #1918(timer) KafkaConnect(kafka-connect/pars-kc): Error reconciling connector ccat-connector-source-incr
io.strimzi.operator.cluster.operator.assembly.ConnectRestException: GET /connectors/ccat-connector-source-incr/config returned 500 (Internal Server Error): Request timed out
        at io.strimzi.operator.cluster.operator.assembly.KafkaConnectApiImpl.lambda$doGet$6(KafkaConnectApiImpl.java:138) ~[io.strimzi.cluster-operator-0.35.0.jar:0.35.0]
        at io.vertx.core.impl.future.FutureImpl$1.onSuccess(FutureImpl.java:91) ~[io.vertx.vertx-core-4.3.8.jar:4.3.8]
        at io.vertx.core.impl.future.FutureBase.lambda$emitSuccess$0(FutureBase.java:54) ~[io.vertx.vertx-core-4.3.8.jar:4.3.8]
        at io.vertx.core.impl.EventLoopContext.execute(EventLoopContext.java:86) ~[io.vertx.vertx-core-4.3.8.jar:4.3.8]
        at io.vertx.core.impl.ContextBase.execute(ContextBase.java:225) ~[io.vertx.vertx-core-4.3.8.jar:4.3.8]
        at io.vertx.core.impl.future.FutureBase.emitSuccess(FutureBase.java:51) ~[io.vertx.vertx-core-4.3.8.jar:4.3.8]
        at io.vertx.core.impl.future.FutureImpl.tryComplete(FutureImpl.java:211) ~[io.vertx.vertx-core-4.3.8.jar:4.3.8]
        at io.vertx.core.impl.future.PromiseImpl.tryComplete(PromiseImpl.java:23) ~[io.vertx.vertx-core-4.3.8.jar:4.3.8]
        at io.vertx.core.http.impl.HttpEventHandler.handleEnd(HttpEventHandler.java:79) ~[io.vertx.vertx-core-4.3.8.jar:4.3.8]
        at io.vertx.core.http.impl.HttpClientResponseImpl.handleEnd(HttpClientResponseImpl.java:250) ~[io.vertx.vertx-core-4.3.8.jar:4.3.8]
        at io.vertx.core.http.impl.Http1xClientConnection$StreamImpl.lambda$new$0(Http1xClientConnection.java:435) ~[io.vertx.vertx-core-4.3.8.jar:4.3.8]
        at io.vertx.core.streams.impl.InboundBuffer.handleEvent(InboundBuffer.java:239) ~[io.vertx.vertx-core-4.3.8.jar:4.3.8]
        at io.vertx.core.streams.impl.InboundBuffer.write(InboundBuffer.java:129) ~[io.vertx.vertx-core-4.3.8.jar:4.3.8]
        at io.vertx.core.http.impl.Http1xClientConnection$StreamImpl.handleEnd(Http1xClientConnection.java:682) ~[io.vertx.vertx-core-4.3.8.jar:4.3.8]
        at io.vertx.core.impl.EventLoopContext.execute(EventLoopContext.java:76) ~[io.vertx.vertx-core-4.3.8.jar:4.3.8]
        at io.vertx.core.impl.ContextBase.execute(ContextBase.java:232) ~[io.vertx.vertx-core-4.3.8.jar:4.3.8]
        at io.vertx.core.http.impl.Http1xClientConnection.handleResponseEnd(Http1xClientConnection.java:916) ~[io.vertx.vertx-core-4.3.8.jar:4.3.8]
        at io.vertx.core.http.impl.Http1xClientConnection.handleHttpMessage(Http1xClientConnection.java:788) ~[io.vertx.vertx-core-4.3.8.jar:4.3.8]
        at io.vertx.core.http.impl.Http1xClientConnection.handleMessage(Http1xClientConnection.java:752) ~[io.vertx.vertx-core-4.3.8.jar:4.3.8]
        at io.vertx.core.net.impl.ConnectionBase.read(ConnectionBase.java:157) ~[io.vertx.vertx-core-4.3.8.jar:4.3.8]
        at io.vertx.core.net.impl.VertxHandler.channelRead(VertxHandler.java:153) ~[io.vertx.vertx-core-4.3.8.jar:4.3.8]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) ~[io.netty.netty-transport-4.1.87.Final.jar:4.1.87.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[io.netty.netty-transport-4.1.87.Final.jar:4.1.87.Final]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[io.netty.netty-transport-4.1.87.Final.jar:4.1.87.Final]
        at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:436) ~[io.netty.netty-transport-4.1.87.Final.jar:4.1.87.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) ~[io.netty.netty-codec-4.1.87.Final.jar:4.1.87.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) ~[io.netty.netty-codec-4.1.87.Final.jar:4.1.87.Final]
        at io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:251) ~[io.netty.netty-transport-4.1.87.Final.jar:4.1.87.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) ~[io.netty.netty-transport-4.1.87.Final.jar:4.1.87.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[io.netty.netty-transport-4.1.87.Final.jar:4.1.87.Final]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[io.netty.netty-transport-4.1.87.Final.jar:4.1.87.Final]
        at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:280) ~[io.netty.netty-handler-4.1.87.Final.jar:4.1.87.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) ~[io.netty.netty-transport-4.1.87.Final.jar:4.1.87.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[io.netty.netty-transport-4.1.87.Final.jar:4.1.87.Final]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[io.netty.netty-transport-4.1.87.Final.jar:4.1.87.Final]
        at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[io.netty.netty-transport-4.1.87.Final.jar:4.1.87.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) ~[io.netty.netty-transport-4.1.87.Final.jar:4.1.87.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[io.netty.netty-transport-4.1.87.Final.jar:4.1.87.Final]
        at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[io.netty.netty-transport-4.1.87.Final.jar:4.1.87.Final]
        at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) ~[io.netty.netty-transport-4.1.87.Final.jar:4.1.87.Final]
        at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) ~[io.netty.netty-transport-4.1.87.Final.jar:4.1.87.Final]
        at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) ~[io.netty.netty-transport-4.1.87.Final.jar:4.1.87.Final]
        at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) ~[io.netty.netty-transport-4.1.87.Final.jar:4.1.87.Final]
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) ~[io.netty.netty-transport-4.1.87.Final.jar:4.1.87.Final]
        at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) ~[io.netty.netty-common-4.1.87.Final.jar:4.1.87.Final]
        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[io.netty.netty-common-4.1.87.Final.jar:4.1.87.Final]
        at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[io.netty.netty-common-4.1.87.Final.jar:4.1.87.Final]
        at java.lang.Thread.run(Thread.java:833) ~[?:?]
2023-06-05 15:19:11 WARN  AbstractConnectOperator:944 - Reconciliation #1918(timer) KafkaConnect(kafka-connect/pars-kc): Error reconciling connector artemis-connector
io.strimzi.operator.cluster.operator.assembly.ConnectRestException: GET /connectors/artemis-connector/config returned 500 (Internal Server Error): Request timed out
        at io.strimzi.operator.cluster.operator.assembly.KafkaConnectApiImpl.lambda$doGet$6(KafkaConnectApiImpl.java:138) ~[io.strimzi.cluster-operator-0.35.0.jar:0.35.0]
        at io.vertx.core.impl.future.FutureImpl$1.onSuccess(FutureImpl.java:91) ~[io.vertx.vertx-core-4.3.8.jar:4.3.8]
        at io.vertx.core.impl.future.FutureBase.lambda$emitSuccess$0(FutureBase.java:54) ~[io.vertx.vertx-core-4.3.8.jar:4.3.8]
        at io.vertx.core.impl.EventLoopContext.execute(EventLoopContext.java:86) ~[io.vertx.vertx-core-4.3.8.jar:4.3.8]
        at io.vertx.core.impl.ContextBase.execute(ContextBase.java:225) ~[io.vertx.vertx-core-4.3.8.jar:4.3.8]
        at io.vertx.core.impl.future.FutureBase.emitSuccess(FutureBase.java:51) ~[io.vertx.vertx-core-4.3.8.jar:4.3.8]
        at io.vertx.core.impl.future.FutureImpl.tryComplete(FutureImpl.java:211) ~[io.vertx.vertx-core-4.3.8.jar:4.3.8]
        at io.vertx.core.impl.future.PromiseImpl.tryComplete(PromiseImpl.java:23) ~[io.vertx.vertx-core-4.3.8.jar:4.3.8]
        at io.vertx.core.http.impl.HttpEventHandler.handleEnd(HttpEventHandler.java:79) ~[io.vertx.vertx-core-4.3.8.jar:4.3.8]
        at io.vertx.core.http.impl.HttpClientResponseImpl.handleEnd(HttpClientResponseImpl.java:250) ~[io.vertx.vertx-core-4.3.8.jar:4.3.8]
        at io.vertx.core.http.impl.Http1xClientConnection$StreamImpl.lambda$new$0(Http1xClientConnection.java:435) ~[io.vertx.vertx-core-4.3.8.jar:4.3.8]
        at io.vertx.core.streams.impl.InboundBuffer.handleEvent(InboundBuffer.java:239) ~[io.vertx.vertx-core-4.3.8.jar:4.3.8]
        at io.vertx.core.streams.impl.InboundBuffer.write(InboundBuffer.java:129) ~[io.vertx.vertx-core-4.3.8.jar:4.3.8]
        at io.vertx.core.http.impl.Http1xClientConnection$StreamImpl.handleEnd(Http1xClientConnection.java:682) ~[io.vertx.vertx-core-4.3.8.jar:4.3.8]
        at io.vertx.core.impl.EventLoopContext.execute(EventLoopContext.java:76) ~[io.vertx.vertx-core-4.3.8.jar:4.3.8]
        at io.vertx.core.impl.ContextBase.execute(ContextBase.java:232) ~[io.vertx.vertx-core-4.3.8.jar:4.3.8]
        at io.vertx.core.http.impl.Http1xClientConnection.handleResponseEnd(Http1xClientConnection.java:916) ~[io.vertx.vertx-core-4.3.8.jar:4.3.8]
        at io.vertx.core.http.impl.Http1xClientConnection.handleHttpMessage(Http1xClientConnection.java:788) ~[io.vertx.vertx-core-4.3.8.jar:4.3.8]
        at io.vertx.core.http.impl.Http1xClientConnection.handleMessage(Http1xClientConnection.java:752) ~[io.vertx.vertx-core-4.3.8.jar:4.3.8]
        at io.vertx.core.net.impl.ConnectionBase.read(ConnectionBase.java:157) ~[io.vertx.vertx-core-4.3.8.jar:4.3.8]
        at io.vertx.core.net.impl.VertxHandler.channelRead(VertxHandler.java:153) ~[io.vertx.vertx-core-4.3.8.jar:4.3.8]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) ~[io.netty.netty-transport-4.1.87.Final.jar:4.1.87.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[io.netty.netty-transport-4.1.87.Final.jar:4.1.87.Final]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[io.netty.netty-transport-4.1.87.Final.jar:4.1.87.Final]
        at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:436) ~[io.netty.netty-transport-4.1.87.Final.jar:4.1.87.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) ~[io.netty.netty-codec-4.1.87.Final.jar:4.1.87.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) ~[io.netty.netty-codec-4.1.87.Final.jar:4.1.87.Final]
        at io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:251) ~[io.netty.netty-transport-4.1.87.Final.jar:4.1.87.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) ~[io.netty.netty-transport-4.1.87.Final.jar:4.1.87.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[io.netty.netty-transport-4.1.87.Final.jar:4.1.87.Final]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[io.netty.netty-transport-4.1.87.Final.jar:4.1.87.Final]
        at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:280) ~[io.netty.netty-handler-4.1.87.Final.jar:4.1.87.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) ~[io.netty.netty-transport-4.1.87.Final.jar:4.1.87.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[io.netty.netty-transport-4.1.87.Final.jar:4.1.87.Final]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[io.netty.netty-transport-4.1.87.Final.jar:4.1.87.Final]
        at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[io.netty.netty-transport-4.1.87.Final.jar:4.1.87.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) ~[io.netty.netty-transport-4.1.87.Final.jar:4.1.87.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[io.netty.netty-transport-4.1.87.Final.jar:4.1.87.Final]
        at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[io.netty.netty-transport-4.1.87.Final.jar:4.1.87.Final]
        at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) ~[io.netty.netty-transport-4.1.87.Final.jar:4.1.87.Final]
        at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) ~[io.netty.netty-transport-4.1.87.Final.jar:4.1.87.Final]
        at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) ~[io.netty.netty-transport-4.1.87.Final.jar:4.1.87.Final]
        at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) ~[io.netty.netty-transport-4.1.87.Final.jar:4.1.87.Final]
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) ~[io.netty.netty-transport-4.1.87.Final.jar:4.1.87.Final]
        at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) ~[io.netty.netty-common-4.1.87.Final.jar:4.1.87.Final]
        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[io.netty.netty-common-4.1.87.Final.jar:4.1.87.Final]
        at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[io.netty.netty-common-4.1.87.Final.jar:4.1.87.Final]
        at java.lang.Thread.run(Thread.java:833) ~[?:?]
2023-06-05 15:19:11 WARN  AbstractConnectOperator:944 - Reconciliation #1918(timer) KafkaConnect(kafka-connect/pars-kc): Error reconciling connector ccat-connector-sink
io.strimzi.operator.cluster.operator.assembly.ConnectRestException: GET /connectors/ccat-connector-sink/config returned 500 (Internal Server Error): Request timed out
        at io.strimzi.operator.cluster.operator.assembly.KafkaConnectApiImpl.lambda$doGet$6(KafkaConnectApiImpl.java:138) ~[io.strimzi.cluster-operator-0.35.0.jar:0.35.0]
        at io.vertx.core.impl.future.FutureImpl$1.onSuccess(FutureImpl.java:91) ~[io.vertx.vertx-core-4.3.8.jar:4.3.8]
        at io.vertx.core.impl.future.FutureBase.lambda$emitSuccess$0(FutureBase.java:54) ~[io.vertx.vertx-core-4.3.8.jar:4.3.8]
        at io.vertx.core.impl.EventLoopContext.execute(EventLoopContext.java:86) ~[io.vertx.vertx-core-4.3.8.jar:4.3.8]
        at io.vertx.core.impl.ContextBase.execute(ContextBase.java:225) ~[io.vertx.vertx-core-4.3.8.jar:4.3.8]
        at io.vertx.core.impl.future.FutureBase.emitSuccess(FutureBase.java:51) ~[io.vertx.vertx-core-4.3.8.jar:4.3.8]
        at io.vertx.core.impl.future.FutureImpl.tryComplete(FutureImpl.java:211) ~[io.vertx.vertx-core-4.3.8.jar:4.3.8]
        at io.vertx.core.impl.future.PromiseImpl.tryComplete(PromiseImpl.java:23) ~[io.vertx.vertx-core-4.3.8.jar:4.3.8]
        at io.vertx.core.http.impl.HttpEventHandler.handleEnd(HttpEventHandler.java:79) ~[io.vertx.vertx-core-4.3.8.jar:4.3.8]
        at io.vertx.core.http.impl.HttpClientResponseImpl.handleEnd(HttpClientResponseImpl.java:250) ~[io.vertx.vertx-core-4.3.8.jar:4.3.8]
        at io.vertx.core.http.impl.Http1xClientConnection$StreamImpl.lambda$new$0(Http1xClientConnection.java:435) ~[io.vertx.vertx-core-4.3.8.jar:4.3.8]
        at io.vertx.core.streams.impl.InboundBuffer.handleEvent(InboundBuffer.java:239) ~[io.vertx.vertx-core-4.3.8.jar:4.3.8]
        at io.vertx.core.streams.impl.InboundBuffer.write(InboundBuffer.java:129) ~[io.vertx.vertx-core-4.3.8.jar:4.3.8]
        at io.vertx.core.http.impl.Http1xClientConnection$StreamImpl.handleEnd(Http1xClientConnection.java:682) ~[io.vertx.vertx-core-4.3.8.jar:4.3.8]
        at io.vertx.core.impl.EventLoopContext.execute(EventLoopContext.java:76) ~[io.vertx.vertx-core-4.3.8.jar:4.3.8]
        at io.vertx.core.impl.ContextBase.execute(ContextBase.java:232) ~[io.vertx.vertx-core-4.3.8.jar:4.3.8]
        at io.vertx.core.http.impl.Http1xClientConnection.handleResponseEnd(Http1xClientConnection.java:916) ~[io.vertx.vertx-core-4.3.8.jar:4.3.8]
        at io.vertx.core.http.impl.Http1xClientConnection.handleHttpMessage(Http1xClientConnection.java:788) ~[io.vertx.vertx-core-4.3.8.jar:4.3.8]
        at io.vertx.core.http.impl.Http1xClientConnection.handleMessage(Http1xClientConnection.java:752) ~[io.vertx.vertx-core-4.3.8.jar:4.3.8]
        at io.vertx.core.net.impl.ConnectionBase.read(ConnectionBase.java:157) ~[io.vertx.vertx-core-4.3.8.jar:4.3.8]
        at io.vertx.core.net.impl.VertxHandler.channelRead(VertxHandler.java:153) ~[io.vertx.vertx-core-4.3.8.jar:4.3.8]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) ~[io.netty.netty-transport-4.1.87.Final.jar:4.1.87.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[io.netty.netty-transport-4.1.87.Final.jar:4.1.87.Final]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[io.netty.netty-transport-4.1.87.Final.jar:4.1.87.Final]
        at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:436) ~[io.netty.netty-transport-4.1.87.Final.jar:4.1.87.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) ~[io.netty.netty-codec-4.1.87.Final.jar:4.1.87.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) ~[io.netty.netty-codec-4.1.87.Final.jar:4.1.87.Final]
        at io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:251) ~[io.netty.netty-transport-4.1.87.Final.jar:4.1.87.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) ~[io.netty.netty-transport-4.1.87.Final.jar:4.1.87.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[io.netty.netty-transport-4.1.87.Final.jar:4.1.87.Final]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[io.netty.netty-transport-4.1.87.Final.jar:4.1.87.Final]
        at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:280) ~[io.netty.netty-handler-4.1.87.Final.jar:4.1.87.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) ~[io.netty.netty-transport-4.1.87.Final.jar:4.1.87.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[io.netty.netty-transport-4.1.87.Final.jar:4.1.87.Final]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[io.netty.netty-transport-4.1.87.Final.jar:4.1.87.Final]
        at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[io.netty.netty-transport-4.1.87.Final.jar:4.1.87.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) ~[io.netty.netty-transport-4.1.87.Final.jar:4.1.87.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[io.netty.netty-transport-4.1.87.Final.jar:4.1.87.Final]
        at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[io.netty.netty-transport-4.1.87.Final.jar:4.1.87.Final]
        at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) ~[io.netty.netty-transport-4.1.87.Final.jar:4.1.87.Final]
        at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) ~[io.netty.netty-transport-4.1.87.Final.jar:4.1.87.Final]
        at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) ~[io.netty.netty-transport-4.1.87.Final.jar:4.1.87.Final]
        at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) ~[io.netty.netty-transport-4.1.87.Final.jar:4.1.87.Final]
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) ~[io.netty.netty-transport-4.1.87.Final.jar:4.1.87.Final]
        at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) ~[io.netty.netty-common-4.1.87.Final.jar:4.1.87.Final]
        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[io.netty.netty-common-4.1.87.Final.jar:4.1.87.Final]
        at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[io.netty.netty-common-4.1.87.Final.jar:4.1.87.Final]
        at java.lang.Thread.run(Thread.java:833) ~[?:?]
2023-06-05 15:19:11 WARN  AbstractConnectOperator:944 - Reconciliation #1918(timer) KafkaConnect(kafka-connect/pars-kc): Error reconciling connector artemis-connector-sink
io.strimzi.operator.cluster.operator.assembly.ConnectRestException: GET /connectors/artemis-connector-sink/config returned 500 (Internal Server Error): Request timed out
        at io.strimzi.operator.cluster.operator.assembly.KafkaConnectApiImpl.lambda$doGet$6(KafkaConnectApiImpl.java:138) ~[io.strimzi.cluster-operator-0.35.0.jar:0.35.0]
        at io.vertx.core.impl.future.FutureImpl$1.onSuccess(FutureImpl.java:91) ~[io.vertx.vertx-core-4.3.8.jar:4.3.8]
        at io.vertx.core.impl.future.FutureBase.lambda$emitSuccess$0(FutureBase.java:54) ~[io.vertx.vertx-core-4.3.8.jar:4.3.8]
        at io.vertx.core.impl.EventLoopContext.execute(EventLoopContext.java:86) ~[io.vertx.vertx-core-4.3.8.jar:4.3.8]
        at io.vertx.core.impl.ContextBase.execute(ContextBase.java:225) ~[io.vertx.vertx-core-4.3.8.jar:4.3.8]
        at io.vertx.core.impl.future.FutureBase.emitSuccess(FutureBase.java:51) ~[io.vertx.vertx-core-4.3.8.jar:4.3.8]
        at io.vertx.core.impl.future.FutureImpl.tryComplete(FutureImpl.java:211) ~[io.vertx.vertx-core-4.3.8.jar:4.3.8]
        at io.vertx.core.impl.future.PromiseImpl.tryComplete(PromiseImpl.java:23) ~[io.vertx.vertx-core-4.3.8.jar:4.3.8]
        at io.vertx.core.http.impl.HttpEventHandler.handleEnd(HttpEventHandler.java:79) ~[io.vertx.vertx-core-4.3.8.jar:4.3.8]
        at io.vertx.core.http.impl.HttpClientResponseImpl.handleEnd(HttpClientResponseImpl.java:250) ~[io.vertx.vertx-core-4.3.8.jar:4.3.8]
        at io.vertx.core.http.impl.Http1xClientConnection$StreamImpl.lambda$new$0(Http1xClientConnection.java:435) ~[io.vertx.vertx-core-4.3.8.jar:4.3.8]
        at io.vertx.core.streams.impl.InboundBuffer.handleEvent(InboundBuffer.java:239) ~[io.vertx.vertx-core-4.3.8.jar:4.3.8]
        at io.vertx.core.streams.impl.InboundBuffer.write(InboundBuffer.java:129) ~[io.vertx.vertx-core-4.3.8.jar:4.3.8]
        at io.vertx.core.http.impl.Http1xClientConnection$StreamImpl.handleEnd(Http1xClientConnection.java:682) ~[io.vertx.vertx-core-4.3.8.jar:4.3.8]
        at io.vertx.core.impl.EventLoopContext.execute(EventLoopContext.java:76) ~[io.vertx.vertx-core-4.3.8.jar:4.3.8]
        at io.vertx.core.impl.ContextBase.execute(ContextBase.java:232) ~[io.vertx.vertx-core-4.3.8.jar:4.3.8]
        at io.vertx.core.http.impl.Http1xClientConnection.handleResponseEnd(Http1xClientConnection.java:916) ~[io.vertx.vertx-core-4.3.8.jar:4.3.8]
        at io.vertx.core.http.impl.Http1xClientConnection.handleHttpMessage(Http1xClientConnection.java:788) ~[io.vertx.vertx-core-4.3.8.jar:4.3.8]
        at io.vertx.core.http.impl.Http1xClientConnection.handleMessage(Http1xClientConnection.java:752) ~[io.vertx.vertx-core-4.3.8.jar:4.3.8]
        at io.vertx.core.net.impl.ConnectionBase.read(ConnectionBase.java:157) ~[io.vertx.vertx-core-4.3.8.jar:4.3.8]
        at io.vertx.core.net.impl.VertxHandler.channelRead(VertxHandler.java:153) ~[io.vertx.vertx-core-4.3.8.jar:4.3.8]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) ~[io.netty.netty-transport-4.1.87.Final.jar:4.1.87.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[io.netty.netty-transport-4.1.87.Final.jar:4.1.87.Final]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[io.netty.netty-transport-4.1.87.Final.jar:4.1.87.Final]
        at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:436) ~[io.netty.netty-transport-4.1.87.Final.jar:4.1.87.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) ~[io.netty.netty-codec-4.1.87.Final.jar:4.1.87.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) ~[io.netty.netty-codec-4.1.87.Final.jar:4.1.87.Final]
        at io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:251) ~[io.netty.netty-transport-4.1.87.Final.jar:4.1.87.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) ~[io.netty.netty-transport-4.1.87.Final.jar:4.1.87.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[io.netty.netty-transport-4.1.87.Final.jar:4.1.87.Final]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[io.netty.netty-transport-4.1.87.Final.jar:4.1.87.Final]
        at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:280) ~[io.netty.netty-handler-4.1.87.Final.jar:4.1.87.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) ~[io.netty.netty-transport-4.1.87.Final.jar:4.1.87.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[io.netty.netty-transport-4.1.87.Final.jar:4.1.87.Final]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[io.netty.netty-transport-4.1.87.Final.jar:4.1.87.Final]
        at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[io.netty.netty-transport-4.1.87.Final.jar:4.1.87.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) ~[io.netty.netty-transport-4.1.87.Final.jar:4.1.87.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[io.netty.netty-transport-4.1.87.Final.jar:4.1.87.Final]
        at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[io.netty.netty-transport-4.1.87.Final.jar:4.1.87.Final]
        at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) ~[io.netty.netty-transport-4.1.87.Final.jar:4.1.87.Final]
        at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) ~[io.netty.netty-transport-4.1.87.Final.jar:4.1.87.Final]
        at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) ~[io.netty.netty-transport-4.1.87.Final.jar:4.1.87.Final]
        at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) ~[io.netty.netty-transport-4.1.87.Final.jar:4.1.87.Final]
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) ~[io.netty.netty-transport-4.1.87.Final.jar:4.1.87.Final]
        at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) ~[io.netty.netty-common-4.1.87.Final.jar:4.1.87.Final]
        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[io.netty.netty-common-4.1.87.Final.jar:4.1.87.Final]
        at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[io.netty.netty-common-4.1.87.Final.jar:4.1.87.Final]
        at java.lang.Thread.run(Thread.java:833) ~[?:?]
2023-06-05 15:19:11 INFO  CrdOperator:133 - Reconciliation #1918(timer) KafkaConnect(kafka-connect/pars-kc): Status of KafkaConnector ccat-connector-source-incr in namespace kafka-connect has been updated
2023-06-05 15:19:11 INFO  CrdOperator:133 - Reconciliation #1918(timer) KafkaConnect(kafka-connect/pars-kc): Status of KafkaConnector artemis-connector in namespace kafka-connect has been updated
2023-06-05 15:19:11 INFO  CrdOperator:133 - Reconciliation #1918(timer) KafkaConnect(kafka-connect/pars-kc): Status of KafkaConnector ccat-connector-sink in namespace kafka-connect has been updated
2023-06-05 15:19:11 INFO  CrdOperator:133 - Reconciliation #1918(timer) KafkaConnect(kafka-connect/pars-kc): Status of KafkaConnector artemis-connector-sink in namespace kafka-connect has been updated
2023-06-05 15:19:11 INFO  CrdOperator:133 - Reconciliation #1918(timer) KafkaConnect(kafka-connect/pars-kc): Status of KafkaConnect pars-kc in namespace kafka-connect has been updated
2023-06-05 15:19:11 INFO  AbstractOperator:510 - Reconciliation #1918(timer) KafkaConnect(kafka-connect/pars-kc): reconciled
2023-06-05 15:19:32 ERROR AbstractOperator:492 - Watcher closed with exception in namespace kafka-connect
io.fabric8.kubernetes.client.WatcherException: Could not process websocket message
        at io.fabric8.kubernetes.client.dsl.internal.WatcherWebSocketListener.onError(WatcherWebSocketListener.java:51) ~[io.fabric8.kubernetes-client-6.6.1.jar:?]
        at io.fabric8.kubernetes.client.jdkhttp.JdkWebSocketImpl$ListenerAdapter.onError(JdkWebSocketImpl.java:85) ~[io.fabric8.kubernetes-httpclient-jdk-6.6.1.jar:?]
        at jdk.internal.net.http.websocket.WebSocketImpl$ReceiveTask.processError(WebSocketImpl.java:508) ~[java.net.http:?]
        at jdk.internal.net.http.websocket.WebSocketImpl$ReceiveTask.run(WebSocketImpl.java:462) ~[java.net.http:?]
        at jdk.internal.net.http.common.SequentialScheduler$CompleteRestartableTask.run(SequentialScheduler.java:149) ~[java.net.http:?]
        at jdk.internal.net.http.common.SequentialScheduler$SchedulableTask.run(SequentialScheduler.java:230) ~[java.net.http:?]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?]
        at java.lang.Thread.run(Thread.java:833) ~[?:?]
Caused by: java.net.SocketException: Connection reset
        at sun.nio.ch.SocketChannelImpl.throwConnectionReset(SocketChannelImpl.java:394) ~[?:?]
        at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:426) ~[?:?]
        at jdk.internal.net.http.SocketTube.readAvailable(SocketTube.java:1170) ~[java.net.http:?]
        at jdk.internal.net.http.SocketTube$InternalReadPublisher$InternalReadSubscription.read(SocketTube.java:833) ~[java.net.http:?]
        at jdk.internal.net.http.SocketTube$SocketFlowTask.run(SocketTube.java:181) ~[java.net.http:?]
        at jdk.internal.net.http.common.SequentialScheduler$SchedulableTask.run(SequentialScheduler.java:230) ~[java.net.http:?]
        at jdk.internal.net.http.common.SequentialScheduler.runOrSchedule(SequentialScheduler.java:303) ~[java.net.http:?]
        at jdk.internal.net.http.common.SequentialScheduler.runOrSchedule(SequentialScheduler.java:256) ~[java.net.http:?]
        at jdk.internal.net.http.SocketTube$InternalReadPublisher$InternalReadSubscription.signalReadable(SocketTube.java:774) ~[java.net.http:?]
        at jdk.internal.net.http.SocketTube$InternalReadPublisher$ReadEvent.signalEvent(SocketTube.java:957) ~[java.net.http:?]
        at jdk.internal.net.http.SocketTube$SocketFlowEvent.handle(SocketTube.java:253) ~[java.net.http:?]
        at jdk.internal.net.http.HttpClientImpl$SelectorManager.handleEvent(HttpClientImpl.java:979) ~[java.net.http:?]
        at jdk.internal.net.http.HttpClientImpl$SelectorManager.lambda$run$3(HttpClientImpl.java:934) ~[java.net.http:?]
        at java.util.ArrayList.forEach(ArrayList.java:1511) ~[?:?]
        at jdk.internal.net.http.HttpClientImpl$SelectorManager.run(HttpClientImpl.java:934) ~[java.net.http:?]
2023-06-05 15:20:48 INFO  ClusterOperator:139 - Triggering periodic reconciliation for namespace kafka-connect
2023-06-05 15:20:48 WARN  StatusUtils:151 - Reconciliation #1922(timer) KafkaConnect(kafka-connect/pars-kc): Contains object at path spec.build.plugins.artifacts with an unknown property: version
2023-06-05 15:20:48 WARN  StatusUtils:151 - Reconciliation #1922(timer) KafkaConnect(kafka-connect/pars-kc): Contains object at path spec.build.plugins.artifacts with an unknown property: version
2023-06-05 15:20:48 INFO  AbstractOperator:239 - Reconciliation #1922(timer) KafkaConnect(kafka-connect/pars-kc): KafkaConnect pars-kc will be checked for creation or modification
2023-06-05 15:20:48 WARN  StatusUtils:151 - Reconciliation #1923(timer) KafkaConnect(kafka-connect/ccat): Contains object at path spec.build.plugins.artifacts with an unknown property: version
2023-06-05 15:20:48 WARN  StatusUtils:151 - Reconciliation #1923(timer) KafkaConnect(kafka-connect/ccat): Contains object at path spec.build.plugins.artifacts with an unknown property: version
2023-06-05 15:20:48 INFO  AbstractOperator:239 - Reconciliation #1923(timer) KafkaConnect(kafka-connect/ccat): KafkaConnect ccat will be checked for creation or modification
2023-06-05 15:20:49 INFO  AbstractConnectOperator:502 - Reconciliation #1923(timer) KafkaConnect(kafka-connect/ccat): creating/updating connector: ccat-connector-source-bulk
2023-06-05 15:20:49 INFO  AbstractConnectOperator:502 - Reconciliation #1923(timer) KafkaConnect(kafka-connect/ccat): creating/updating connector: ccat-sink-connector
2023-06-05 15:20:49 INFO  AbstractOperator:510 - Reconciliation #1923(timer) KafkaConnect(kafka-connect/ccat): reconciled
2023-06-05 15:21:48 INFO  AbstractOperator:380 - Reconciliation #1922(timer) KafkaConnect(kafka-connect/pars-kc): Reconciliation is in progress
2023-06-05 15:22:48 INFO  ClusterOperator:139 - Triggering periodic reconciliation for namespace kafka-connect
2023-06-05 15:22:48 WARN  StatusUtils:151 - Reconciliation #1925(timer) KafkaConnect(kafka-connect/ccat): Contains object at path spec.build.plugins.artifacts with an unknown property: version
2023-06-05 15:22:48 WARN  StatusUtils:151 - Reconciliation #1925(timer) KafkaConnect(kafka-connect/ccat): Contains object at path spec.build.plugins.artifacts with an unknown property: version
2023-06-05 15:22:48 INFO  AbstractOperator:239 - Reconciliation #1925(timer) KafkaConnect(kafka-connect/ccat): KafkaConnect ccat will be checked for creation or modification
2023-06-05 15:22:48 INFO  AbstractOperator:380 - Reconciliation #1922(timer) KafkaConnect(kafka-connect/pars-kc): Reconciliation is in progress
2023-06-05 15:22:49 INFO  AbstractConnectOperator:502 - Reconciliation #1925(timer) KafkaConnect(kafka-connect/ccat): creating/updating connector: ccat-connector-source-bulk
2023-06-05 15:22:49 INFO  AbstractConnectOperator:502 - Reconciliation #1925(timer) KafkaConnect(kafka-connect/ccat): creating/updating connector: ccat-sink-connector
2023-06-05 15:22:49 INFO  AbstractOperator:510 - Reconciliation #1925(timer) KafkaConnect(kafka-connect/ccat): reconciled

Lots of reconciliation errors in operator logs or websocket issues

Strimzi

KafkaConnect Pending State/CrashLoopBackOffs #8608

Uh oh!

Uh oh!

imrohankataria Jun 4, 2023

Replies: 1 comment · 14 replies

Uh oh!

scholzj Jun 4, 2023 Maintainer

Uh oh!

imrohankataria Jun 5, 2023 Author

Uh oh!

imrohankataria Jun 5, 2023 Author

Uh oh!

imrohankataria Jun 5, 2023 Author

Uh oh!

scholzj Jun 5, 2023 Maintainer

Uh oh!

imrohankataria Jun 5, 2023 Author

Uh oh!

scholzj Jun 5, 2023 Maintainer

Uh oh!

imrohankataria Jun 5, 2023 Author

Uh oh!

Uh oh!

imrohankataria Jun 5, 2023 Author

imrohankataria
Jun 4, 2023

Replies: 1 comment 14 replies

scholzj
Jun 4, 2023
Maintainer

imrohankataria Jun 5, 2023
Author

imrohankataria Jun 5, 2023
Author

imrohankataria Jun 5, 2023
Author

scholzj Jun 5, 2023
Maintainer

imrohankataria Jun 5, 2023
Author

scholzj Jun 5, 2023
Maintainer

imrohankataria Jun 5, 2023
Author

imrohankataria Jun 5, 2023
Author