Upgrading from 0.28 to 0.32

[sreejesh-radhakrishnan-db]

Hi I have a working cluster which run on 0.28 and I am trying to upgrade to 0.32 ( and kafka 3.3.1).

I am using HELM based install - and Operator upgrade has gone through well. and I can see the operator running in 0.32.0

but I can see this error on Operator log

io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: GET at: https://240.2.134.193/apis/coordination.k8s.io/v1/namespaces/kafka/leases/strimzi-cluster-operator. Message: Forbidden!Configured service account doesn't have access. Service account may have been revoked. leases.coordination.k8s.io "strimzi-cluster-operator" is forbidden: User "system:serviceaccount:kafka:strimzi-cluster-operator" cannot get resource "leases" in API group "coordination.k8s.io" in the namespace "kafka".

and
textPayload: "2023-01-11 16:12:41 ERROR LeaderElector:156 - Exception occurred while acquiring lock 'LeaseLock: kafka - strimzi-cluster-operator (strimzi-cluster-operator-6b9bddd689-4lssq)'"
timestamp: "2023-01-11T16:12:41.068330167Z"

I checked the SA (strimzi-cluster-operator) is present with role binding done as below

roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: strimzi-cluster-operator-namespaced
subjects:

• kind: ServiceAccount
  name: strimzi-cluster-operator
  namespace: myproject

clusterROle strimzi-cluster-operator-namespaced is also present which I checked and it seems to be matching with what is there in your repo (which was expected anyways).

what am I missing ? As far the document goes 0.28 -> 0.32 should be straright fwd upgrade isnt?

[scholzj]

I guess you are missing the RBAC for the Lease resource for leader election. But as I do not use Helm, no idea why.

[scholzj]

In YAML files, it is covered by these two files: https://github.com/strimzi/strimzi-kafka-operator/blob/main/install/cluster-operator/022-ClusterRole-strimzi-cluster-operator-role.yaml and https://github.com/strimzi/strimzi-kafka-operator/blob/main/install/cluster-operator/022-RoleBinding-strimzi-cluster-operator.yaml

[sreejesh-radhakrishnan-db]

Interestingly I can see this is been done as well, I can see this in GKE.

roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: strimzi-cluster-operator-leader-election
subjects:

• kind: ServiceAccount
  name: strimzi-cluster-operator
  namespace: myproject

[sreejesh-radhakrishnan-db]

I go it, name space is wrong

[amitmiglani91]

can you please guide me step that you follow to upgrade strimzi

[sreejesh123]

For any setup all you need to do is , pull new helm chart as documented in strimzi upgrade section and then upgrade operator and then Kafka.

just to add , I ended up with issue as we copy the Crd from GitHub ( than to pull from helm repo of strimzi) due to some limitation in our setup . And whist doing that I didn’t give enough notice to namespace for two new yaml added, the upgrade is smooth except for one Kafka cluster, which I am still struggling to find and answer. Posted a separate q for that.

[amitmiglani91]

Thanq so much @sreejesh123

[amitmiglani91]

@sreejesh-radhakrishnan-db can you Please hare cmds that you follow to upgrade. Strimzi and kafka using helm. that will be really helpful form me.

[sreejesh-radhakrishnan-db]

SO I use helm install I use the helm command , I have the artifacts in my local repo and I refer to that and values.yaml as follows

      helm upgrade strimzi-dev-kafka-operator ./helmcharts/strimzi-kafka-operator-0.32.0 -i -n ${KAFKA_NAMESPACE} \
       --debug

helm upgrade gpds-kafka-broker \
       ./helmcharts/kafka-cluster-setup-3.1.0 -i -n ${KAFKA_NAMESPACE} --debug \
        --set "kafka_cluster_name=${KAFKA_CLUSTER_NAME}" \
        --set "env=${ENV}"  --set "kafka.version=3.3.1"  

the tempalte which you downlaod from strimxzi need to be edited to override the generic namespace hardcoded from "kafka " to anythihing specific to your usecase.

all above is explained in strimzi upgrade doc.

[sreejesh-radhakrishnan-db]

@scholzj - in one of the KAKFA cluster I can see this ERROR and upgrade fails - with ZOOKEEPER failing to come back up

2023-01-29 13:28:14.771 GMT
2023-01-29 13:28:14 DEBUG CustomResource:165 - Calling CustomResource#setApiVersion doesn't do anything because the API version is computed and shouldn't be changed
2023-01-29 13:28:14.772 GMT
2023-01-29 13:28:14 DEBUG CustomResource:176 - Calling CustomResource#setKind doesn't do anything because the Kind is computed and shouldn't be changed

Sceanrio- RUN kafka manifest update

Strimzi log says above thing repeatedly.

ZK-0 vanishes and ZK-1 and ZK-2 goes on a NPE error with unable to contact ZK-0 .

any idea what is happening?

I am updating from 0.28 to .32

[scholzj]

These messages can be ignored:

2023-01-29 13:28:14 DEBUG CustomResource:165 - Calling CustomResource#setApiVersion doesn't do anything because the API version is computed and shouldn't be changed
2023-01-29 13:28:14 DEBUG CustomResource:176 - Calling CustomResource#setKind doesn't do anything because the Kind is computed and shouldn't be changed

They do not mean anything bad.

[scholzj]

I think you will need to share the full logs and CRs.

[sreejesh-radhakrishnan-db]

now I have downgraded to 0.28, let me get the log from timeframe and uplaod. will Strimzi Operator log be enough?

[scholzj]

I have no idea what the problem is, so I'm afraid I have no clue what exactly would be needed.

[sreejesh-radhakrishnan-db]

@scholzj I will uplaod all the log isnide the kafka namespace as soon as I get compliance approval to upload the log. mean time I have a q

for those cluster which I could upgrade succesfully I wanted a clarification.

for 0.28 I can see KAKFA and ZK running a statefuset and then inside they are managing three POD (as per my replicaiton need)

But on 0.32 I am not able to see any stateful set, but I can see three KAKFA POD and ZK running.

Is this behaviour correct? I thought as per what I saw in one of your work item discussion you are not removing statefulset till 0.35. hence thought to clarify

$ kubectl get statefulset -n dbca-kafka
No resources found.

$ kubectl get pods -n dbca-kafka
NAME READY STATUS RESTARTS AGE
config-reloader-reloader-7dd4566578-ggzzg 1/1 Running 0 19d
confluen-schema-registry-5d998cf856-z2x5p 1/1 Running 0 17d
dbca-cluster-entity-operator-6d8fb8bc5-tvt2k 3/3 Running 1 (3d2h ago) 3d2h
dbca-cluster-kafka-0 1/1 Running 0 2d14h
dbca-cluster-kafka-1 1/1 Running 0 2d14h
dbca-cluster-kafka-2 1/1 Running 0 2d14h
dbca-cluster-kafka-exporter-5d8c54bd8b-d25d6 1/1 Running 0 3d2h
dbca-cluster-zookeeper-0 1/1 Running 0 3d2h
dbca-cluster-zookeeper-1 1/1 Running 0 3d2h
dbca-cluster-zookeeper-2 1/1 Running 0 3d2h
strimzi-cluster-operator-968b87999-k6mmf 1/1 Running 0 3d2h

---

Version of PODS
8 artifactory.sdlc.ctl.gcp.db.com/dkr-all/com/db/opensourcecicd/strimzi/kafka:0.32.0-kafka-3.3.1.1
3 artifactory.sdlc.ctl.gcp.db.com/dkr-all/dkr-quay/strimzi/operator:0.32.0

[scholzj]

StatefulSets will be removed in 0.35. But already for some time Strimzi not using them by default and uses StrimziPodSets instead. You would need to specifically enable them through the feature gates configuration.

[sreejesh-radhakrishnan-db]

@scholzj finally I for complaince approval to uplaod log, its been uploded as a CSV> please filter on COLOUMN "resource.labels.container_name" to see log from differenct component. Sorry I could get any other better ways to extract logs. as the POD was recreted afterfailure to upgrade. and taking this log from GKE log than K8s log.

[sreejesh-radhakrishnan-db]

strimzi_log_upload.csv

[sreejesh-radhakrishnan-db]

@scholzj I did one more try today where I upgrade operator 0.32 , then uninstalled kafka 3.3.1 which is not supported on 0.32. then tried an upgrade.

the kakfa SVC are created on 240.xxx range on 9092, but there are no POD associated with that SVC.

Attaching the log. again its on CSV, but all I have is Operator log with CONNECITON error during reconcile
downloaded-logs-20230208-183445.csv

[scholzj]

Why is no pod associated with it? Are they running? Do they match the label selector?

[sreejesh-radhakrishnan-db]

I waited for almost 40min. NO POD started other than just the service on the namespace

[scholzj]

The service is always created first. So you need to investigate what happens with the pods. DEBUG logs from the operator might help, Kubernetes Events and logs might help you.

[sreejesh-radhakrishnan-db]

@scholzj thank you. I will get debug log and have a look on k8s detail log. AFAIK no event or K8s alert or warning log was generated.

[sreejesh-radhakrishnan-db]

this is only two ERROR I can see on K8s

gcp-controller-manager failed to be retrieved
8 February 2023 at 20:46:13 GMT+0
Resource name
core/v1/namespaces/kube-system/services/gcp-controller-manager
Error message
Not found (HTTP 404): Not Found

and

certificatesigningrequests failed to be read
8 February 2023 at 20:46:13 GMT+0
Resource name
certificates.k8s.io/v1beta1/certificatesigningrequests
Error message
Not found (HTTP 404): Not Found
is any related to Strimzi you reckon?

[scholzj]

I don't know anything about your environment and have no logs. So it's hard to say what is related or not. This does not seem like it. The complete operator log at DEBUG level should be really the start.

[sreejesh-radhakrishnan-db]

attaching the debug log as requestd,

my setup

GKE - private cluster 1.23.14-gke.1800
three ZONE cluster with kafka setup with ZONE aware
Uploading downloaded-logs-20230208-211047.csv…

[scholzj]

Looks like the upload failed. Also, can you provide the log as a regular log file? I cannot really read the CSV file easily.

[sreejesh-radhakrishnan-db]

uplaoded as txt

[sreejesh-radhakrishnan-db]

upgradelog.txt

[scholzj]

Right, so ... you need to figure out why this is happening:

2023-02-08 18:26:29 INFO  StrimziPodSetController:292 - Reconciliation #310(watch) StrimziPodSet(gpds-kafka/gpds-dev1-zookeeper): StrimziPodSet will be reconciled
2023-02-08 18:26:40 ERROR StrimziPodSetController:323 - Reconciliation #310(watch) StrimziPodSet(gpds-kafka/gpds-dev1-zookeeper): StrimziPodSet gpds-dev1-zookeeper in namespace gpds-kafka reconciliation failed
"io.fabric8.kubernetes.client.KubernetesClientException: An error has occurred.
	at io.fabric8.kubernetes.client.KubernetesClientException.launderThrowable(KubernetesClientException.java:129) ~[io.fabric8.kubernetes-client-api-6.2.0.jar:?]
	at io.fabric8.kubernetes.client.KubernetesClientException.launderThrowable(KubernetesClientException.java:122) ~[io.fabric8.kubernetes-client-api-6.2.0.jar:?]
	at io.fabric8.kubernetes.client.dsl.internal.CreateOnlyResourceOperation.create(CreateOnlyResourceOperation.java:44) ~[io.fabric8.kubernetes-client-6.2.0.jar:?]
	at io.fabric8.kubernetes.client.dsl.internal.BaseOperation.create(BaseOperation.java:1073) ~[io.fabric8.kubernetes-client-6.2.0.jar:?]
	at io.fabric8.kubernetes.client.dsl.internal.BaseOperation.create(BaseOperation.java:88) ~[io.fabric8.kubernetes-client-6.2.0.jar:?]
	at io.strimzi.operator.cluster.operator.assembly.StrimziPodSetController.maybeCreateOrPatchPod(StrimziPodSetController.java:384) ~[io.strimzi.cluster-operator-0.32.0.jar:0.32.0]
	at io.strimzi.operator.cluster.operator.assembly.StrimziPodSetController.reconcile(StrimziPodSetController.java:312) ~[io.strimzi.cluster-operator-0.32.0.jar:0.32.0]
	at io.strimzi.operator.cluster.operator.assembly.StrimziPodSetController.run(StrimziPodSetController.java:480) ~[io.strimzi.cluster-operator-0.32.0.jar:0.32.0]
	at java.lang.Thread.run(Thread.java:829) ~[?:?]
Caused by: java.io.IOException: timeout
	at io.fabric8.kubernetes.client.dsl.internal.OperationSupport.waitForResult(OperationSupport.java:514) ~[io.fabric8.kubernetes-client-6.2.0.jar:?]
	at io.fabric8.kubernetes.client.dsl.internal.OperationSupport.handleResponse(OperationSupport.java:551) ~[io.fabric8.kubernetes-client-6.2.0.jar:?]
	at io.fabric8.kubernetes.client.dsl.internal.OperationSupport.handleResponse(OperationSupport.java:535) ~[io.fabric8.kubernetes-client-6.2.0.jar:?]
	at io.fabric8.kubernetes.client.dsl.internal.OperationSupport.handleCreate(OperationSupport.java:328) ~[io.fabric8.kubernetes-client-6.2.0.jar:?]
	at io.fabric8.kubernetes.client.dsl.internal.BaseOperation.handleCreate(BaseOperation.java:675) ~[io.fabric8.kubernetes-client-6.2.0.jar:?]
	at io.fabric8.kubernetes.client.dsl.internal.BaseOperation.handleCreate(BaseOperation.java:88) ~[io.fabric8.kubernetes-client-6.2.0.jar:?]
	at io.fabric8.kubernetes.client.dsl.internal.CreateOnlyResourceOperation.create(CreateOnlyResourceOperation.java:42) ~[io.fabric8.kubernetes-client-6.2.0.jar:?]
	... 6 more
Caused by: java.net.SocketTimeoutException: timeout
	at okhttp3.internal.http2.Http2Stream$StreamTimeout.newTimeoutException(Http2Stream.java:672) ~[com.squareup.okhttp3.okhttp-3.12.12.jar:?]
	at okhttp3.internal.http2.Http2Stream$StreamTimeout.exitAndThrowIfTimedOut(Http2Stream.java:680) ~[com.squareup.okhttp3.okhttp-3.12.12.jar:?]
	at okhttp3.internal.http2.Http2Stream.takeHeaders(Http2Stream.java:153) ~[com.squareup.okhttp3.okhttp-3.12.12.jar:?]
	at okhttp3.internal.http2.Http2Codec.readResponseHeaders(Http2Codec.java:131) ~[com.squareup.okhttp3.okhttp-3.12.12.jar:?]
	at okhttp3.internal.http.CallServerInterceptor.intercept(CallServerInterceptor.java:88) ~[com.squareup.okhttp3.okhttp-3.12.12.jar:?]
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147) ~[com.squareup.okhttp3.okhttp-3.12.12.jar:?]
	at okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.java:45) ~[com.squareup.okhttp3.okhttp-3.12.12.jar:?]
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147) ~[com.squareup.okhttp3.okhttp-3.12.12.jar:?]
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121) ~[com.squareup.okhttp3.okhttp-3.12.12.jar:?]
	at okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.java:93) ~[com.squareup.okhttp3.okhttp-3.12.12.jar:?]
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147) ~[com.squareup.okhttp3.okhttp-3.12.12.jar:?]
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121) ~[com.squareup.okhttp3.okhttp-3.12.12.jar:?]
	at okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.java:93) ~[com.squareup.okhttp3.okhttp-3.12.12.jar:?]
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147) ~[com.squareup.okhttp3.okhttp-3.12.12.jar:?]
	at okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.java:127) ~[com.squareup.okhttp3.okhttp-3.12.12.jar:?]
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147) ~[com.squareup.okhttp3.okhttp-3.12.12.jar:?]
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121) ~[com.squareup.okhttp3.okhttp-3.12.12.jar:?]
	at io.fabric8.kubernetes.client.okhttp.OkHttpClientBuilderImpl$InteceptorAdapter.intercept(OkHttpClientBuilderImpl.java:70) ~[io.fabric8.kubernetes-httpclient-okhttp-6.2.0.jar:?]
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147) ~[com.squareup.okhttp3.okhttp-3.12.12.jar:?]
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121) ~[com.squareup.okhttp3.okhttp-3.12.12.jar:?]
	at io.fabric8.kubernetes.client.okhttp.OkHttpClientBuilderImpl$InteceptorAdapter.intercept(OkHttpClientBuilderImpl.java:70) ~[io.fabric8.kubernetes-httpclient-okhttp-6.2.0.jar:?]
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147) ~[com.squareup.okhttp3.okhttp-3.12.12.jar:?]
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121) ~[com.squareup.okhttp3.okhttp-3.12.12.jar:?]
	at io.fabric8.kubernetes.client.okhttp.OkHttpClientBuilderImpl$InteceptorAdapter.intercept(OkHttpClientBuilderImpl.java:70) ~[io.fabric8.kubernetes-httpclient-okhttp-6.2.0.jar:?]
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147) ~[com.squareup.okhttp3.okhttp-3.12.12.jar:?]
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121) ~[com.squareup.okhttp3.okhttp-3.12.12.jar:?]
	at okhttp3.RealCall.getResponseWithInterceptorChain(RealCall.java:257) ~[com.squareup.okhttp3.okhttp-3.12.12.jar:?]
	at okhttp3.RealCall$AsyncCall.execute(RealCall.java:201) ~[com.squareup.okhttp3.okhttp-3.12.12.jar:?]
	at okhttp3.internal.NamedRunnable.run(NamedRunnable.java:32) ~[com.squareup.okhttp3.okhttp-3.12.12.jar:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
	... 1 more"
2023-02-08 18:26:40 INFO  StrimziPodSetController:328 - Reconciliation #310(watch) StrimziPodSet(gpds-kafka/gpds-dev1-zookeeper): reconciled

This will be the request to create or patch the pod which is failing. So you need to find out why is your Kube cluster letting it timeout while it seems to work fine for other clusters. Could be some webhook or something doing something wrong for example or some other tooling which is running in your cluster.

[sreejesh-radhakrishnan-db]

sure, I will raise a ticket with Google too, as in my K8s cluser LOG ther e is nothing I can infer to this error. BTW StrimziPodSet is this something which need some special permission to work?

[scholzj]

The call which is failing is likely just a Pod creation call - so nothing special.

[sreejesh-radhakrishnan-db]

Understood. I can see two webhook and the POD creation is ofcourse timing out. You in past said we can enable STAEFULSET instead of STRIMZIPODSET if we change a config, will you be kind enbough to point me to that page please?

I am trying to see if those two webhook is causing the issue. One webhook is associated with NEWRELIC and one with SPARK-OPERATOR> my money is on SPARK OPERATOR cauing some funny stuff and POD creation is failing.

I am doing multiple installs , tought if I know how to change to stateulset, I can try to see if that goes through or not.

[scholzj]

Noz at my desk right now. So I cannot share the link. But looking for UseStrimziPodSet feature gate in the docs you will find iit. That said, StatefulSets will not be supported forever, so it is just a temporary workaround.

…

On Thu, Feb 9, 2023, 11:59 sree ***@***.***> wrote: Understood. I can see two webhook and the POD creation is ofcourse timing out. You in past said we can enable STAEFULSET instead of STRIMZIPODSET if we change a config, will you be kind enbough to point me to that page please? I am trying to see if those two webhook is causing the issue. One webhook is associated with NEWRELIC and one with SPARK-OPERATOR> my money is on SPARK OPERATOR cauing some funny stuff and POD creation is failing. I am doing multiple installs , tought if I know how to change to stateulset, I can try to see if that goes through or not. — Reply to this email directly, view it on GitHub <#7895 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABLFORZSDS6SLH4U3ZFMM63WWTEYFANCNFSM6AAAAAATYHK2O4> . You are receiving this because you were mentioned.Message ID: <strimzi/strimzi-kafka-operator/repo-discussions/7895/comments/4917498@ github.com>

[sreejesh-radhakrishnan-db]

agreed. and thank you

[sreejesh-radhakrishnan-db]

Interesting @scholzj install went just fine with STATEFULSET change. Now I am trying to install with STRIMZIPODSET without the SPARKOPERATOR to see if that will go okay or not.

[sreejesh-radhakrishnan-db]

Unfortuante STRIMZIPODSET when I use the POD creation is still failing for the WEBHOOK timing out. Not sure is this something which @scholzj can address or not? I.e why STRIMZIPODSET on admission webhook failure timeout on POD creation?

some finding

Call to API master from log: Unfortunate only way I can download is as a JSON or CSV. Both you dont like personally. SO I am trying to put in basic bit I found interesting. Please let me know

authorization.k8s.io/reason - RBAC: allowed by RoleBinding "strimzi-cluster-operator/gpds-kafka" of ClusterRole "strimzi-cluster-operator-namespaced" to ServiceAccount "strimzi-cluster-operator/gpds-kafka"

protoPayload.authorizationInfo.permission - ["io.k8s.core.v1.pods.create"]

protoPayload.authorizationInfo.resource - ["core/v1/namespaces/gpds-kafka/pods/gpds-dev1-zookeeper-0"]

protoPayload.methodName - io.k8s.core.v1.pods.create

protoPayload.request.@type
core.k8s.io/v1.Pod

protoPayload.request.metadata.labels."app.kubernetes.io/managed-by" :strimzi-cluster-operator

protoPayload.request.spec.containers :
[{"name":"zookeeper","livenessProbe":{"failureThreshold":3,"timeoutSeconds":5,"exec":{"command":["/opt/kafka/zookeeper_healthcheck.sh"]},"periodSeconds":10,"initialDelaySeconds":15,"successThreshold":1},"imagePullPolicy":"IfNotPresent","env":[{"value":"true","name":"ZOOKEEPER_METRICS_ENABLED"},{"value":"true","name":"ZOOKEEPER_SNAPSHOT_CHECK_ENABLED"},{"name":"STRIMZI_KAFKA_GC_LOG_ENABLED","value":"false"},{"name":"ZOOKEEPER_JMX_ENABLED","value":"true"},{"value":"-Xms128M","name":"KAFKA_HEAP_OPTS"},{"value":"tickTime=2000\ninitLimit=5\nsyncLimit=2\nautopurge.purgeInterval=1\nadmin.enableServer=false\n","name":"ZOOKEEPER_CONFIGURATION"}],"terminationMessagePath":"/dev/termination-log","image":"artifactory.sdlc.ctl.gcp.db.com/dkr-all/com/db/opensourcecicd/strimzi/kafka:0.32.0-kafka-3.3.1.1","command":["/opt/kafka/zookeeper_run.sh"],"volumeMounts":[{"mountPath":"/tmp","name":"strimzi-tmp"},{"name":"data","mountPath":"/var/lib/zookeeper"},{"mountPath":"/opt/kafka/custom-config/","name":"zookeeper-metrics-and-logging"},{"name":"zookeeper-nodes","mountPath":"/opt/kafka/zookeeper-node-certs/"},{"mountPath":"/opt/kafka/cluster-ca-certs/","name":"cluster-ca-certs"}],"resources":{},"ports":[{"containerPort":2888,"protocol":"TCP","name":"tcp-clustering"},{"name":"tcp-election","containerPort":3888,"protocol":"TCP"},{"containerPort":2181,"protocol":"TCP","name":"tcp-clients"},{"containerPort":9404,"name":"tcp-prometheus","protocol":"TCP"},{"protocol":"TCP","containerPort":9999,"name":"jmx"}],"readinessProbe":{"initialDelaySeconds":15,"periodSeconds":10,"successThreshold":1,"failureThreshold":3,"exec":{"command":["/opt/kafka/zookeeper_healthcheck.sh"]},"timeoutSeconds":5},"terminationMessagePolicy":"File"}]

protoPayload.response.message : Timeout: request did not complete within requested timeout - context canceled

respnse code - 504

protoPayload.status.message :Gateway Timeout

labels: {
authorization.k8s.io/decision: "allow"
authorization.k8s.io/reason: "RBAC: allowed by RoleBinding "strimzi-cluster-operator/gpds-kafka" of ClusterRole "strimzi-cluster-operator-namespaced" to ServiceAccount "strimzi-cluster-operator/gpds-kafka""
failed-open.mutation.webhook.admission.k8s.io/round_0_index_7: "metadata-injection.newrelic.com"
mutation.webhook.admission.k8s.io/round_0_index_7: "{"configuration":"newrelic-nri-metadata-injection","webhook":"metadata-injection.newrelic.com","mutated":false}"
}

errorjson.log

[scholzj]

TBH, I do not know where this log is coming from or what is the gateway that times out. This is your cluster, so I'm afraid you would need to investigate it

[sreejesh-radhakrishnan-db]

@scholzj We use GKE and we use NewRelic workload for logging and monitoring. ( so as to coexist between on-prem and cloud. )

issue is when STRIMZI or any workload request to create a POD we have a mutating WEBHOOK which get triggered which add some env specific variable to new POD created. in my case whist installing Strimzi 0.32 with StrimziPodSet this is where the connection timeout is happening.

And this mutating webhook is only failing when we install as StrimziPodSet and Not when I install as just normal or K8S STATEFULSET by overriding the operator/crd/manifest using UseStrimziPodSet .

We cannot promote 0.32 to UAT or PRD, unless we know what is causing the webhook to timeout whist using the default UseStrimziPodSet,

to debug further will it be possible for me to see the code where you are creating UseStrimziPodSet how its different to normal k8s statefulset? I may struggle to understand but will get google support help if needed.

Not sure if there is any restriction in StrimziPodSet - that WEBHOOK is not allowed or something in that line is what Google also inferring?

Any help would be much appreciated

[scholzj]

TBH, I understand that you might not be fully in control. But the webhook should give you a reason why it is not allowing the pod. Reverse engineering it from the other side would be super hard. So to be honest, whoever is supporting those tools in your cluster should give you the answers. And based on them we can see if this is something we can address in Strimzi or not. If they are unable to do that, I think such tools is dangerous and might shutdown your production for an unknown reason at any time.

It is quite hard to point you to a single place in our code where to look at. The StrimziPodSetController class is creating the Pods. But how they look is defined through many different parts of the Cluster operator. The KafkaCluster class will be generating some of it, but methods from the ZooKeeperReconciler or KafkaReconciler and various util classes will be part of ti as well.

[chandans21j]

@sreejesh-radhakrishnan-db

To supporting only one version of kafka 3.3.1 in kafka-versions.yaml file , do we need to remove another version like 3.2.0,3.2.13.2.3? Actually when i am removing so that time getting error.