Zookeeper Stuck in CrashBackloopoff status When Installing in Redhat 8.6 Working fine on Redat 7.9

[fauzankhanak]

We are using Strimzi kafka on redhat 7.9 and it is working fine but when we are trying to install the same on 8.6 Zookeeper is stuck in crashloopbackoff status.

Docker version : 19.03.15
K8s version: v1.19.9
Redhat Version: Red Hat Enterprise Linux release 8.6 (Ootpa)
Operator version: 0.27.1
kafka version: 0.27.1-kafka-3.0.0

$$kubectl describe pod zookeeper-0 -n kafka
"Events:
Type Reason Age From Message

---

Normal Scheduled 16m default-scheduler Successfully assigned kafka/atomiq-cluster-zookeeper-0 to atomiqplatformrhel8601-vm
Warning Unhealthy 15m kubelet Readiness probe failed:
Warning Unhealthy 15m kubelet Liveness probe failed:
Normal Killing 14m kubelet Container zookeeper failed liveness probe, will be restarted
Normal Created 14m (x4 over 16m) kubelet Created container zookeeper
Normal Pulled 14m (x4 over 16m) kubelet Container image "atomiqplatformrhel8601-vm:7443/strimzi/kafka:0.27.1-kafka-3.0.0" already present on machine
Normal Started 14m (x4 over 16m) kubelet Started container zookeeper
Warning Unhealthy 14m (x3 over 16m) kubelet Readiness probe failed: Ncat: Connection refused.
Warning Unhealthy 14m (x3 over 16m) kubelet Liveness probe failed: Ncat: Connection refused.
Warning BackOff 83s (x58 over 15m) kubelet Back-off restarting failed container"

$$kubectl logs zookeeper-0 -n kafka
2022-11-29 07:52:23,063 ERROR Couldn't bind to atomiq-cluster-zookeeper-0.atomiq-cluster-zookeeper-nodes.kafka.svc:2888 (org.apache.zookeeper.server.quorum.Leader) [QuorumPeermyid=1(secure=0.0.0.0:2181)]
java.net.SocketException: Unresolved address
at java.base/java.net.ServerSocket.bind(ServerSocket.java:388)
at java.base/java.net.ServerSocket.bind(ServerSocket.java:349)
at org.apache.zookeeper.server.quorum.Leader.createServerSocket(Leader.java:315)
at org.apache.zookeeper.server.quorum.Leader.lambda$new$0(Leader.java:294)
at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:195)
at java.base/java.util.concurrent.ConcurrentHashMap$KeySpliterator.forEachRemaining(ConcurrentHashMap.java:3566)
at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484)
at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474)
at java.base/java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:150)
at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:173)
at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
at java.base/java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:497)
at org.apache.zookeeper.server.quorum.Leader.(Leader.java:297)
at org.apache.zookeeper.server.quorum.QuorumPeer.makeLeader(QuorumPeer.java:1272)
at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:1479)
2022-11-29 07:52:23,208 ERROR Leaving listener thread for address atomiq-cluster-zookeeper-0.atomiq-cluster-zookeeper-nodes.kafka.svc:3888 after 3 errors. Use zookeeper.electionPortBindRetry property to increase retry count. (org.apache.zookeeper.server.quorum.QuorumCnxManager) [ListenerHandler-atomiq-cluster-zookeeper-0.atomiq-cluster-zookeeper-nodes.kafka.svc:3888]
2022-11-29 07:52:23,209 ERROR As I'm leaving the listener thread, I won't be able to participate in leader election any longer: atomiq-cluster-zookeeper-0.atomiq-cluster-zookeeper-nodes.kafka.svc:3888 (org.apache.zookeeper.server.quorum.QuorumCnxManager) [QuorumPeerListener]
2022-11-29 07:52:23,210 ERROR Exiting JVM with code 14 (org.apache.zookeeper.util.ServiceUtils) [QuorumPeerListener

I have checked the issues related to this error I know this is related to dns issue, but we have checked all the dns setting we didn't find anything related to dns.

Kube-flannel logs:
I1129 06:18:36.102150 1 main.go:514] Determining IP address of default interface
I1129 06:18:36.103234 1 main.go:527] Using interface with name eth0 and address 10.67.39.73
I1129 06:18:36.103265 1 main.go:544] Defaulting external address to interface address (10.67.39.73)
I1129 06:18:36.201993 1 kube.go:126] Waiting 10m0s for node controller to sync
I1129 06:18:36.202043 1 kube.go:309] Starting kube subnet manager
I1129 06:18:37.202166 1 kube.go:133] Node controller sync successful
I1129 06:18:37.202209 1 main.go:244] Created subnet manager: Kubernetes Subnet Manager - atomiqplatformrhel8601-vm
I1129 06:18:37.202214 1 main.go:247] Installing signal handlers
I1129 06:18:37.202322 1 main.go:386] Found network config - Backend type: vxlan
I1129 06:18:37.202427 1 vxlan.go:120] VXLAN config: VNI=1 Port=0 GBP=false DirectRouting=false
I1129 06:18:37.204109 1 main.go:317] Wrote subnet file to /run/flannel/subnet.env
I1129 06:18:37.204125 1 main.go:321] Running backend.
I1129 06:18:37.204152 1 main.go:339] Waiting for all goroutines to exit
I1129 06:18:37.204171 1 vxlan_network.go:60] watching for new subnet leases

Kube-proxy logs.
W1129 05:49:26.754768 1 proxier.go:649] Failed to load kernel module nf_conntrack_ipv4 with modprobe. You can ignore this message when kube-proxy is running inside container without mounting /lib/modules
I1129 05:49:26.878377 1 node.go:136] Successfully retrieved node IP: 10.67.39.73
I1129 05:49:26.878419 1 server_others.go:142] kube-proxy node IP is an IPv4 address (10.67.39.73), assume IPv4 operation
W1129 05:49:27.085025 1 server_others.go:578] Unknown proxy mode "", assuming iptables proxy
I1129 05:49:27.085155 1 server_others.go:185] Using iptables Proxier.
I1129 05:49:27.085496 1 server.go:650] Version: v1.19.9
I1129 05:49:27.086000 1 conntrack.go:52] Setting nf_conntrack_max to 262144
I1129 05:49:27.086090 1 conntrack.go:100] Set sysctl 'net/netfilter/nf_conntrack_tcp_timeout_established' to 86400
I1129 05:49:27.086131 1 conntrack.go:100] Set sysctl 'net/netfilter/nf_conntrack_tcp_timeout_close_wait' to 3600
I1129 05:49:27.086443 1 config.go:315] Starting service config controller
I1129 05:49:27.086454 1 shared_informer.go:240] Waiting for caches to sync for service config
I1129 05:49:27.086475 1 config.go:224] Starting endpoint slice config controller
I1129 05:49:27.086479 1 shared_informer.go:240] Waiting for caches to sync for endpoint slice config
I1129 05:49:27.186686 1 shared_informer.go:247] Caches are synced for endpoint slice config
I1129 05:49:27.186731 1 shared_informer.go:247] Caches are synced for service config

lsmod o/p
Module Size Used by
iptable_nat 16384 1
iptable_filter 16384 1
ip_tables 28672 2 iptable_filter,iptable_nat
xt_statistic 16384 3
vxlan 65536 0
ip6_udp_tunnel 16384 1 vxlan
udp_tunnel 20480 1 vxlan
ipt_REJECT 16384 8
nf_reject_ipv4 16384 1 ipt_REJECT
ip_vs_sh 16384 0
ip_vs_wrr 16384 0
ip_vs_rr 16384 0
ip_vs 172032 6 ip_vs_rr,ip_vs_sh,ip_vs_wrr
xt_comment 16384 214
xt_mark 16384 6
xt_nat 16384 43
veth 28672 0
ipt_MASQUERADE 16384 11
nf_conntrack_netlink 49152 0
xt_addrtype 16384 17
nft_chain_nat 16384 12
nf_nat 45056 4 ipt_MASQUERADE,xt_nat,nft_chain_nat,iptable_nat
br_netfilter 24576 0
bridge 278528 1 br_netfilter
stp 16384 1 bridge
llc 16384 2 bridge,stp
nft_counter 16384 278
xt_conntrack 16384 11
nf_conntrack 172032 6 xt_conntrack,nf_nat,ipt_MASQUERADE,xt_nat,nf_conntrack_netlink,ip_vs
nf_defrag_ipv6 20480 2 nf_conntrack,ip_vs
nf_defrag_ipv4 16384 1 nf_conntrack
xt_owner 16384 1
nft_compat 20480 453
overlay 139264 152
nf_tables 180224 1047 nft_compat,nft_counter,nft_chain_nat
nfnetlink 16384 4 nft_compat,nf_conntrack_netlink,nf_tables
vfat 20480 1
fat 81920 1 vfat
intel_rapl_msr 16384 0
intel_rapl_common 24576 1 intel_rapl_msr
isst_if_mbox_msr 16384 0
isst_if_common 16384 1 isst_if_mbox_msr
kvm_intel 339968 0
kvm 905216 1 kvm_intel
irqbypass 16384 1 kvm
crct10dif_pclmul 16384 1
crc32_pclmul 16384 0
ghash_clmulni_intel 16384 0
rapl 20480 0
hv_utils 45056 1
hv_balloon 32768 0
hyperv_fb 24576 1
i2c_piix4 24576 0
ata_generic 16384 0
pcspkr 16384 0
joydev 24576 0
binfmt_misc 20480 1
xfs 1556480 11
libcrc32c 16384 5 nf_conntrack,nf_nat,nf_tables,xfs,ip_vs
sd_mod 53248 12
t10_pi 16384 1 sd_mod
sg 40960 0
hv_netvsc 94208 0
hv_storvsc 24576 8
scsi_transport_fc 81920 1 hv_storvsc
hid_hyperv 16384 0
hyperv_keyboard 16384 0
ata_piix 36864 0
libata 262144 2 ata_piix,ata_generic
crc32c_intel 24576 1
hv_vmbus 131072 7 hv_balloon,hv_utils,hv_netvsc,hid_hyperv,hv_storvsc,hyperv_keyboard,hyperv_fb
serio_raw 16384 0
dm_mirror 28672 0
dm_region_hash 20480 1 dm_mirror
dm_log 20480 2 dm_region_hash,dm_mirror
dm_mod 151552 32 dm_log,dm_mirror
ipmi_devintf 20480 0
ipmi_msghandler 110592 1 ipmi_devintf

Is there any specific kernel setting we have to do in redhat 8.6 please suggest.

[scholzj]

This sounds to me like some infrastructure issue and not like a Strimzi issue. But I never saw an issue like this. I have seen issues with the Pods talking with each other due to DNS or firewall issues. Bu not being unable to bind to the port. Maybe you need to check the security contexts, privileges of the container engine etc.?