Multiple ztunnel getting stuck in crashloopbackoff with error wrokloadmanager failed to start proxy: can't assign requested address

[karan56625]

Hi folks, we are running istio 1.28 on very high load AKS cluster. Recently we on-boarded Istio on the cluster and put logging components under service mesh, we saw 4-5 ztunnel pods getting crashedloopbaxkoff with the following error:
"
2026-01-07T17:51:33.579503Z error inpod::workloadmanager failed to start proxy: Bind([::1]:15053, Os { code: 99, kind: AddrNotAvailable, message: "Cannot assign requested address" }) uid=2a6140db-b27a-4f11-a7ff-58643b6ce31f
"
Due to this, workload inside the servicemesh as well got crashedloopbackoff.
Even restarting the ztunnel didn't help there. Once we put the workload outside the mesh, both ztunnels and workload pods started running fine.

Istio; 1.28
Platform: AKS 1.32 with Azure linux 3.0

[Stevenjin8]

@karan56625 my first hunch is that this has something to do with ipv4 vs ipv6. Are you running IPv6 on your cluster?

[Stevenjin8]

Some other things,

1. could you look at ztunnel's logs and check for a message that looks something like:

failed to determine if IPv6 was disabled; continuing anyways, but this may fail.

2. could you show the ztunnel deployment?
3. finally, can you try to change you're deployment, but add the IPV6_ENABLED="false" environment variable?

[karan56625]

Hi @Stevenjin8 ,

1. Ztunnel logs:

2026-01-07T11:25:42.118977Z	info	ztunnel	version: version.BuildInfo{Version:"1.28.0", GitRevision:"unknown", RustVersion:"1.89.0", BuildProfile:"release", BuildStatus:"Modified", IstioVersion:"${VERSION}", CryptoProvider:"tls-boring"}	
2026-01-07T11:25:42.119018Z	info	ztunnel	set file descriptor limits from 1048576 to 1048576	
2026-01-07T11:25:42.119177Z	info	ztunnel	running with config: proxy: true
dnsProxy: true
windowSize: 4194304
connectionWindowSize: 16777216
frameSize: 1048576
poolMaxStreamsPerConn: 100
poolUnusedReleaseTimeout:
  secs: 300
  nanos: 0
socks5Addr: null
adminAddr: !Localhost
- true
- 15000
statsAddr: !SocketAddr '[::]:15020'
readinessAddr: !SocketAddr '[::]:15021'
inboundAddr: '[::]:15008'
inboundPlaintextAddr: '[::]:15006'
outboundAddr: '[::]:15001'
dnsProxyAddr: !Localhost
- true
- 15053
illegalPorts:
- 15008
- 15001
- 15006
network: ''
localNode: aks-weu320s3z2-37019218-vmss0001th
proxyMode: Shared
proxyWorkloadInformation: null
clusterId: Kubernetes
clusterDomain: cluster.local
caAddress: https://istiod.istio.svc:15012
caRootCert: !File ./var/run/secrets/istio/root-cert.pem
altCaHostname: null
xdsAddress: https://istiod.istio.svc:15012
xdsRootCert: !File ./var/run/secrets/istio/root-cert.pem
altXdsHostname: null
preferedServiceNamespace: null
secretTtl:
  secs: 86400
  nanos: 0
xdsOnDemand: false
fakeCa: false
fakeSelfInbound: false
selfTerminationDeadline:
  secs: 25
  nanos: 0
proxyMetadata:
  ENABLE_HBONE: 'true'
  CLUSTER_ID: Kubernetes
  ISTIO_VERSION: ${VERSION}
numWorkerThreads: 2
requireOriginalSource: null
proxyArgs: proxy ztunnel
dnsResolverCfg:
  domain: null
  search:
  - istio.svc.cluster.local
  - svc.cluster.local
  - cluster.local
  - jgdndtaaxveevbq5s1zn1eiiah.ax.internal.cloudapp.net
  name_servers:
  - socket_addr: 10.0.0.10:53
    protocol: udp
    tls_dns_name: null
    http_endpoint: null
    trust_negative_responses: false
    bind_addr: null
  - socket_addr: 10.0.0.10:53
    protocol: tcp
    tls_dns_name: null
    http_endpoint: null
    trust_negative_responses: false
    bind_addr: null
dnsResolverOpts:
  ndots: 5
  timeout:
    secs: 5
    nanos: 0
  attempts: 2
  check_names: true
  edns0: false
  validate: false
  ip_strategy: Ipv4AndIpv6
  cache_size: 4096
  use_hosts_file: Auto
  positive_min_ttl: null
  negative_min_ttl: null
  positive_max_ttl: null
  negative_max_ttl: null
  num_concurrent_reqs: 2
  preserve_intermediates: true
  try_tcp_on_error: false
  server_ordering_strategy: QueryStatistics
  recursion_desired: true
  avoid_local_udp_ports: []
  os_port_selection: false
  case_randomization: false
  trust_anchor: null
inpodUds: /var/run/ztunnel/ztunnel.sock
inpodPortReuse: true
packetMark: 1337
socketConfig:
  keepaliveTime:
    secs: 180
    nanos: 0
  keepaliveInterval:
    secs: 180
    nanos: 0
  keepaliveRetries: 9
  keepaliveEnabled: true
  userTimeoutEnabled: false
xdsHeaders: []
caHeaders: []
localhostAppTunnel: true
ztunnelIdentity: spiffe://cluster.local/ns/istio/sa/ztunnel
ztunnelWorkload:
  name: ztunnel-495m6
  namespace: istio
  serviceAccount: ztunnel
ipv6Enabled: true
	
2026-01-07T11:25:42.119729Z	info	hyper_util	listener established	address=[::]:15021 component="readiness"
2026-01-07T11:25:42.119785Z	info	app	shared proxy mode - in-pod mode enabled	
2026-01-07T11:25:42.119796Z	info	proxyfactory	creating ztunnel self-proxy listener with identity: Spiffe { trust_domain: "cluster.local", namespace: "istio", service_account: "ztunnel" }	
2026-01-07T11:25:42.119832Z	info	proxy::inbound	listener established	address=[::]:15008 component="inbound" transparent=true
2026-01-07T11:25:42.119881Z	info	app	Starting ztunnel inbound listener task	
2026-01-07T11:25:42.119928Z	info	hyper_util	listener established	address=127.0.0.1:15000 component="admin"
2026-01-07T11:25:42.119957Z	info	hyper_util	listener established	address=[::]:15020 component="stats"
2026-01-07T11:25:42.119973Z	info	readiness	Task 'dns proxy' complete (1.535594ms), still awaiting 3 tasks	
2026-01-07T11:25:42.119976Z	info	readiness	Task 'proxy' complete (1.538339ms), still awaiting 2 tasks	
2026-01-07T11:25:42.145587Z	info	xds::client:xds{id=1}	Stream established	
2026-01-07T11:25:42.149970Z	info	xds::client:xds{id=1}	received response	type_url="type.googleapis.com/istio.workload.Address" size=3824 removes=0
2026-01-07T11:25:42.164166Z	info	xds::client:xds{id=1}	received response	type_url="type.googleapis.com/istio.security.Authorization" size=0 removes=0
2026-01-07T11:25:42.164217Z	info	readiness	Task 'state manager' complete (45.779208ms), still awaiting 1 tasks	
2026-01-07T11:25:42.164313Z	info	inpod::workloadmanager	handling new stream	
2026-01-07T11:25:42.164744Z	info	inpod::statemanager	pod received, starting proxy	uid="a02d6167-d214-433a-9af3-76931c66320d" name="geneva-logger-mdsd-hunting-detections-audit-7f96c4756b-fhjvk" namespace="wdatp-infra-system-logging"
2026-01-07T11:25:42.164970Z	info	dns::server	starting local DNS server	address=localhost:15053 component="dns"
2026-01-07T11:25:42.165144Z	info	proxy::inbound	listener established	address=[::]:15008 component="inbound" transparent=true
2026-01-07T11:25:42.165173Z	info	proxy::inbound_passthrough	listener established	address=[::]:15006 component="inbound plaintext" transparent=true
2026-01-07T11:25:42.165200Z	info	proxy::outbound	listener established	address=[::]:15001 component="outbound" transparent=true
2026-01-07T11:25:42.165369Z	info	inpod::statemanager	pod received, starting proxy	uid="0fa94b23-7d27-40ad-b8b3-7e472bb189e7" name="geneva-logger-tvm-7d84d85d66-qc8d2" namespace="wdatp-infra-system-logging"
2026-01-07T11:25:42.165538Z	info	dns::server	starting local DNS server	address=localhost:15053 component="dns"
2026-01-07T11:25:42.165633Z	error	inpod::workloadmanager	failed to start proxy: Bind([::1]:15053, Os { code: 99, kind: AddrNotAvailable, message: "Cannot assign requested address" })	uid=0fa94b23-7d27-40ad-b8b3-7e472bb189e7
2026-01-07T11:25:42.165647Z	info	inpod::workloadmanager	scheduling retry	uids=["0fa94b23-7d27-40ad-b8b3-7e472bb189e7"]
2026-01-07T11:25:42.165798Z	info	inpod::statemanager	pod received, starting proxy	uid="0f10adde-d04e-4376-9be2-ba4b0a539931" name="geneva-logger-mdm-detection-c9db59cb7-nzmjq" namespace="wdatp-infra-system-logging"
2026-01-07T11:25:42.166001Z	info	dns::server	starting local DNS server	address=localhost:15053 component="dns"
2026-01-07T11:25:42.166176Z	info	proxy::inbound	listener established	address=[::]:15008 component="inbound" transparent=true
2026-01-07T11:25:42.166219Z	info	proxy::inbound_passthrough	listener established	address=[::]:15006 component="inbound plaintext" transparent=true
2026-01-07T11:25:42.166261Z	info	proxy::outbound	listener established	address=[::]:15001 component="outbound" transparent=true
2026-01-07T11:25:42.166443Z	info	inpod::statemanager	pod received, starting proxy	uid="c4f372dd-fb4d-4383-9356-4aee4282ce93" name="geneva-logger-mdsd-cloud-576c574bb6-qsv94" namespace="wdatp-infra-system-logging"
2026-01-07T11:25:42.166656Z	info	dns::server	starting local DNS server	address=localhost:15053 component="dns"
2026-01-07T11:25:42.166841Z	info	proxy::inbound	listener established	address=[::]:15008 component="inbound" transparent=true
2026-01-07T11:25:42.166881Z	info	proxy::inbound_passthrough	listener established	address=[::]:15006 component="inbound plaintext" transparent=true
2026-01-07T11:25:42.166909Z	info	proxy::outbound	listener established	address=[::]:15001 component="outbound" transparent=true
2026-01-07T11:25:42.167061Z	info	inpod::statemanager	pod received, starting proxy	uid="9053358a-982c-4419-9476-11582ab4fadc" name="geneva-logger-mdm-hunting-detections-656c5ff565-tnqdp" namespace="wdatp-infra-system-logging"
2026-01-07T11:25:42.167224Z	info	dns::server	starting local DNS server	address=localhost:15053 component="dns"
2026-01-07T11:25:42.167365Z	info	proxy::inbound	listener established	address=[::]:15008 component="inbound" transparent=true
2026-01-07T11:25:42.167394Z	info	proxy::inbound_passthrough	listener established	address=[::]:15006 component="inbound plaintext" transparent=true
2026-01-07T11:25:42.167419Z	info	proxy::outbound	listener established	address=[::]:15001 component="outbound" transparent=true
2026-01-07T11:25:42.167540Z	info	inpod::statemanager	pod received, starting proxy	uid="3d1f46b3-523b-48f3-bd27-eb347dc28726" name="geneva-logger-kube-system-5856ccd854-79frs" namespace="wdatp-infra-system-logging"
2026-01-07T11:25:42.167710Z	info	dns::server	starting local DNS server	address=localhost:15053 component="dns"
2026-01-07T11:25:42.167846Z	info	proxy::inbound	listener established	address=[::]:15008 component="inbound" transparent=true
2026-01-07T11:25:42.167873Z	info	proxy::inbound_passthrough	listener established	address=[::]:15006 component="inbound plaintext" transparent=true
2026-01-07T11:25:42.167898Z	info	proxy::outbound	listener established	address=[::]:15001 component="outbound" transparent=true
2026-01-07T11:25:42.168032Z	info	inpod::statemanager	pod received, starting proxy	uid="4c93e65c-9735-44aa-9415-19f4947bea7e" name="geneva-logger-mdsd-band-6bb4fb9647-6xhxg" namespace="wdatp-infra-system-logging"
2026-01-07T11:25:42.216738Z	info	dns::server	starting local DNS server	address=localhost:15053 component="dns"
2026-01-07T11:25:42.216953Z	info	proxy::inbound	listener established	address=[::]:15008 component="inbound" transparent=true
2026-01-07T11:25:42.216987Z	info	proxy::inbound_passthrough	listener established	address=[::]:15006 component="inbound plaintext" transparent=true
2026-01-07T11:25:42.217016Z	info	proxy::outbound	listener established	address=[::]:15001 component="outbound" transparent=true
2026-01-07T11:25:42.217285Z	info	inpod::statemanager	pod received, starting proxy	uid="2054e106-c890-4bda-971d-7049d087f684" name="geneva-logger-sense-8657c59d6c-544fq" namespace="wdatp-infra-system-logging"
2026-01-07T11:25:42.217541Z	info	dns::server	starting local DNS server	address=localhost:15053 component="dns"
2026-01-07T11:25:42.217693Z	info	proxy::inbound	listener established	address=[::]:15008 component="inbound" transparent=true
2026-01-07T11:25:42.217729Z	info	proxy::inbound_passthrough	listener established	address=[::]:15006 component="inbound plaintext" transparent=true
2026-01-07T11:25:42.217756Z	info	proxy::outbound	listener established	address=[::]:15001 component="outbound" transparent=true
2026-01-07T11:25:42.217952Z	info	inpod::statemanager	pod received, starting proxy	uid="d37885de-9a4e-475f-a6fc-fe4e6dd561ca" name="geneva-logger-mdsd-management-bc5d4ddf4-69vbg" namespace="wdatp-infra-system-logging"
2026-01-07T11:25:42.218123Z	info	dns::server	starting local DNS server	address=localhost:15053 component="dns"
2026-01-07T11:25:42.218283Z	info	proxy::inbound	listener established	address=[::]:15008 component="inbound" transparent=true
2026-01-07T11:25:42.218313Z	info	proxy::inbound_passthrough	listener established	address=[::]:15006 component="inbound plaintext" transparent=true
2026-01-07T11:25:42.218344Z	info	proxy::outbound	listener established	address=[::]:15001 component="outbound" transparent=true
2026-01-07T11:25:42.218532Z	info	inpod::statemanager	pod received, starting proxy	uid="e18c73bf-57ca-4a14-9090-5cbfd98c520a" name="geneva-logger-tvm-7d84d85d66-zvdr6" namespace="wdatp-infra-system-logging"
2026-01-07T11:25:42.218708Z	info	dns::server	starting local DNS server	address=localhost:15053 component="dns"
2026-01-07T11:25:42.218844Z	info	proxy::inbound	listener established	address=[::]:15008 component="inbound" transparent=true
2026-01-07T11:25:42.218872Z	info	proxy::inbound_passthrough	listener established	address=[::]:15006 component="inbound plaintext" transparent=true
2026-01-07T11:25:42.218901Z	info	proxy::outbound	listener established	address=[::]:15001 component="outbound" transparent=true
2026-01-07T11:25:42.219056Z	info	inpod::statemanager	pod received, starting proxy	uid="01c16ef5-55a6-4d72-93f7-a3132509bdd6" name="geneva-logger-m365dsre-56765d6d58-gvw2j" namespace="wdatp-infra-system-logging"
2026-01-07T11:25:42.219228Z	info	dns::server	starting local DNS server	address=localhost:15053 component="dns"
2026-01-07T11:25:42.219358Z	info	proxy::inbound	listener established	address=[::]:15008 component="inbound" transparent=true
2026-01-07T11:25:42.219388Z	info	proxy::inbound_passthrough	listener established	address=[::]:15006 component="inbound plaintext" transparent=true
2026-01-07T11:25:42.219409Z	info	proxy::outbound	listener established	address=[::]:15001 component="outbound" transparent=true
2026-01-07T11:25:42.219522Z	info	inpod::statemanager	pod received, starting proxy	uid="a796725e-dd0d-43b1-a5c2-843b825b11e0" name="geneva-logger-hunting-detections-6587c6cb49-wpgjw" namespace="wdatp-infra-system-logging"
2026-01-07T11:25:42.219706Z	info	dns::server	starting local DNS server	address=localhost:15053 component="dns"
2026-01-07T11:25:42.219838Z	info	proxy::inbound	listener established	address=[::]:15008 component="inbound" transparent=true
2026-01-07T11:25:42.219867Z	info	proxy::inbound_passthrough	listener established	address=[::]:15006 component="inbound plaintext" transparent=true
2026-01-07T11:25:42.219890Z	info	proxy::outbound	listener established	address=[::]:15001 component="outbound" transparent=true
2026-01-07T11:25:42.220007Z	info	inpod::statemanager	pod received, starting proxy	uid="efcf142f-4b9c-4f0f-9707-3b686d8509a4" name="geneva-logger-tvm-7d84d85d66-j4vfc" namespace="wdatp-infra-system-logging"
2026-01-07T11:25:42.220174Z	info	dns::server	starting local DNS server	address=localhost:15053 component="dns"
2026-01-07T11:25:42.220298Z	info	proxy::inbound	listener established	address=[::]:15008 component="inbound" transparent=true
2026-01-07T11:25:42.220325Z	info	proxy::inbound_passthrough	listener established	address=[::]:15006 component="inbound plaintext" transparent=true
2026-01-07T11:25:42.220389Z	info	proxy::outbound	listener established	address=[::]:15001 component="outbound" transparent=true
2026-01-07T11:25:42.220563Z	info	inpod::statemanager	pod received, starting proxy	uid="aae319b7-fc8c-466b-8819-4203328d365e" name="geneva-logger-management-676745bf7d-b2zp6" namespace="wdatp-infra-system-logging"
2026-01-07T11:25:42.220809Z	info	dns::server	starting local DNS server	address=localhost:15053 component="dns"
2026-01-07T11:25:42.220993Z	info	proxy::inbound	listener established	address=[::]:15008 component="inbound" transparent=true
2026-01-07T11:25:42.221023Z	info	proxy::inbound_passthrough	listener established	address=[::]:15006 component="inbound plaintext" transparent=true
2026-01-07T11:25:42.264472Z	info	proxy::outbound	listener established	address=[::]:15001 component="outbound" transparent=true
2026-01-07T11:25:42.266959Z	info	inpod::statemanager	pod received, starting proxy	uid="a3845e9c-b620-4bef-bf0d-05212bf46ec4" name="geneva-logger-mdsd-ndr-847cd599d6-49b8l" namespace="wdatp-infra-system-logging"
2026-01-07T11:25:42.267195Z	info	dns::server	starting local DNS server	address=localhost:15053 component="dns"
2026-01-07T11:25:42.267326Z	info	proxy::inbound	listener established	address=[::]:15008 component="inbound" transparent=true
2026-01-07T11:25:42.267358Z	info	proxy::inbound_passthrough	listener established	address=[::]:15006 component="inbound plaintext" transparent=true
2026-01-07T11:25:42.267380Z	info	proxy::outbound	listener established	address=[::]:15001 component="outbound" transparent=true
2026-01-07T11:25:42.272729Z	info	inpod::statemanager	pod received, starting proxy	uid="a981fc31-30d6-4bb5-9fa7-6596375d5e08" name="geneva-logger-mdm-md-mobile-8686745c7c-4pz69" namespace="wdatp-infra-system-logging"
2026-01-07T11:25:42.273017Z	info	dns::server	starting local DNS server	address=localhost:15053 component="dns"
2026-01-07T11:25:42.273196Z	info	proxy::inbound	listener established	address=[::]:15008 component="inbound" transparent=true
2026-01-07T11:25:42.273242Z	info	proxy::inbound_passthrough	listener established	address=[::]:15006 component="inbound plaintext" transparent=true
2026-01-07T11:25:42.273291Z	info	proxy::outbound	listener established	address=[::]:15001 component="outbound" transparent=true
2026-01-07T11:25:42.274116Z	info	inpod::statemanager	pod received, starting proxy	uid="40019417-daea-40dc-9ce2-6fb8d3b27662" name="geneva-logger-mdsd-mde-response-6f598699d7-dx8n4" namespace="wdatp-infra-system-logging"
2026-01-07T11:25:42.274317Z	info	dns::server	starting local DNS server	address=localhost:15053 component="dns"
2026-01-07T11:25:42.274414Z	error	inpod::workloadmanager	failed to start proxy: Bind([::1]:15053, Os { code: 99, kind: AddrNotAvailable, message: "Cannot assign requested address" })	uid=40019417-daea-40dc-9ce2-6fb8d3b27662
2026-01-07T11:25:42.274938Z	info	inpod::statemanager	pod received, starting proxy	uid="3908c963-81c2-43d1-832d-e62f5b849830" name="geneva-logger-mdm-mde-disruption-5bb6477c6f-9p8fk" namespace="wdatp-infra-system-logging"
2026-01-07T11:25:42.275110Z	info	dns::server	starting local DNS server	address=localhost:15053 component="dns"
2026-01-07T11:25:42.275257Z	info	proxy::inbound	listener established	address=[::]:15008 component="inbound" transparent=true
2026-01-07T11:25:42.275287Z	info	proxy::inbound_passthrough	listener established	address=[::]:15006 component="inbound plaintext" transparent=true
2026-01-07T11:25:42.275310Z	info	proxy::outbound	listener established	address=[::]:15001 component="outbound" transparent=true
2026-01-07T11:25:42.275556Z	info	inpod::statemanager	pod received, starting proxy	uid="37f19c90-0430-42a1-a60d-6a3176a0d6c3" name="geneva-logger-mdm-cloud-68dddd8658-jllf7" namespace="wdatp-infra-system-logging"
2026-01-07T11:25:42.275802Z	info	dns::server	starting local DNS server	address=localhost:15053 component="dns"
2026-01-07T11:25:42.275972Z	info	proxy::inbound	listener established	address=[::]:15008 component="inbound" transparent=true
2026-01-07T11:25:42.276006Z	info	proxy::inbound_passthrough	listener established	address=[::]:15006 component="inbound plaintext" transparent=true
2026-01-07T11:25:42.276083Z	info	proxy::outbound	listener established	address=[::]:15001 component="outbound" transparent=true
2026-01-07T11:25:42.276462Z	info	inpod::statemanager	pod received, starting proxy	uid="0f04e50e-4857-4fbf-b2a0-04a8d2867468" name="geneva-logger-mdm-management-55c4f8b995-lxtml" namespace="wdatp-infra-system-logging"
2026-01-07T11:25:42.276717Z	info	dns::server	starting local DNS server	address=localhost:15053 component="dns"
2026-01-07T11:25:42.276940Z	info	proxy::inbound	listener established	address=[::]:15008 component="inbound" transparent=true
2026-01-07T11:25:42.276985Z	info	proxy::inbound_passthrough	listener established	address=[::]:15006 component="inbound plaintext" transparent=true
2026-01-07T11:25:42.277018Z	info	proxy::outbound	listener established	address=[::]:15001 component="outbound" transparent=true
2026-01-07T11:25:42.277303Z	info	inpod::statemanager	pod received, starting proxy	uid="de35fa01-33ba-4b39-b72c-c9eda0ac41a3" name="node-remediation-node-problem-detector-82gpr" namespace="wdatp-infra-system"
2026-01-07T11:25:42.277651Z	info	dns::server	starting local DNS server	address=localhost:15053 component="dns"
2026-01-07T11:25:42.277879Z	info	proxy::inbound	listener established	address=[::]:15008 component="inbound" transparent=true
2026-01-07T11:25:42.277936Z	info	proxy::inbound_passthrough	listener established	address=[::]:15006 component="inbound plaintext" transparent=true
2026-01-07T11:25:42.277969Z	info	proxy::outbound	listener established	address=[::]:15001 component="outbound" transparent=true
2026-01-07T11:25:42.278172Z	info	inpod::statemanager	pod received, starting proxy	uid="85ace355-8701-4c6a-84b8-d6baa939b4af" name="geneva-logger-mde-disruption-5494985645-d8bvh" namespace="wdatp-infra-system-logging"
2026-01-07T11:25:42.278384Z	info	dns::server	starting local DNS server	address=localhost:15053 component="dns"
2026-01-07T11:25:42.278519Z	info	proxy::inbound	listener established	address=[::]:15008 component="inbound" transparent=true
2026-01-07T11:25:42.278547Z	info	proxy::inbound_passthrough	listener established	address=[::]:15006 component="inbound plaintext" transparent=true
2026-01-07T11:25:42.278579Z	info	proxy::outbound	listener established	address=[::]:15001 component="outbound" transparent=true
2026-01-07T11:25:42.278750Z	info	inpod::statemanager	pod received, starting proxy	uid="4fd44da1-9c7c-45e0-9fd6-67dcc2798753" name="geneva-logger-mdm-ndr-69cc68cf7f-s6zsp" namespace="wdatp-infra-system-logging"
2026-01-07T11:25:42.279002Z	info	dns::server	starting local DNS server	address=localhost:15053 component="dns"
2026-01-07T11:25:42.279198Z	info	proxy::inbound	listener established	address=[::]:15008 component="inbound" transparent=true
2026-01-07T11:25:42.279240Z	info	proxy::inbound_passthrough	listener established	address=[::]:15006 component="inbound plaintext" transparent=true
2026-01-07T11:25:42.279277Z	info	proxy::outbound	listener established	address=[::]:15001 component="outbound" transparent=true
2026-01-07T11:25:42.279461Z	info	inpod::statemanager	pod received, starting proxy	uid="7d9004cc-0890-4029-85f2-e77a6313b941" name="geneva-logger-wdatp-infra-system-logging-5686b8d89b-jpqj6" namespace="wdatp-infra-system-logging"
2026-01-07T11:25:42.279696Z	info	dns::server	starting local DNS server	address=localhost:15053 component="dns"
2026-01-07T11:25:42.279878Z	info	proxy::inbound	listener established	address=[::]:15008 component="inbound" transparent=true
2026-01-07T11:25:42.279920Z	info	proxy::inbound_passthrough	listener established	address=[::]:15006 component="inbound plaintext" transparent=true
2026-01-07T11:25:42.279950Z	info	proxy::outbound	listener established	address=[::]:15001 component="outbound" transparent=true
2026-01-07T11:25:42.280108Z	info	inpod::statemanager	pod received, starting proxy	uid="8f84cee1-e6af-42c4-b09a-d4cb34765fc2" name="geneva-logger-mdm-hunting-detections-656c5ff565-54x2t" namespace="wdatp-infra-system-logging"
2026-01-07T11:25:42.280381Z	info	dns::server	starting local DNS server	address=localhost:15053 component="dns"
2026-01-07T11:25:42.280585Z	info	proxy::inbound	listener established	address=[::]:15008 component="inbound" transparent=true
2026-01-07T11:25:42.280625Z	info	proxy::inbound_passthrough	listener established	address=[::]:15006 component="inbound plaintext" transparent=true
2026-01-07T11:25:42.280655Z	info	proxy::outbound	listener established	address=[::]:15001 component="outbound" transparent=true
2026-01-07T11:25:42.280861Z	info	inpod::statemanager	pod received, starting proxy	uid="1ba7b55b-5dc7-4d06-b3e5-6c68c1f3c2d9" name="geneva-logger-md-mobile-65c9d8f974-nf7qx" namespace="wdatp-infra-system-logging"
2026-01-07T11:25:42.281087Z	info	dns::server	starting local DNS server	address=localhost:15053 component="dns"
2026-01-07T11:25:42.281242Z	info	proxy::inbound	listener established	address=[::]:15008 component="inbound" transparent=true
2026-01-07T11:25:42.324492Z	info	proxy::inbound_passthrough	listener established	address=[::]:15006 component="inbound plaintext" transparent=true
2026-01-07T11:25:42.324548Z	info	proxy::outbound	listener established	address=[::]:15001 component="outbound" transparent=true
2026-01-07T11:25:42.326220Z	info	inpod::statemanager	pod received, starting proxy	uid="9f633c3a-a009-4da2-94b8-864334cb1fb7" name="geneva-logger-mdm-mde-response-556575b99-qxrzn" namespace="wdatp-infra-system-logging"
2026-01-07T11:25:42.326456Z	info	dns::server	starting local DNS server	address=localhost:15053 component="dns"
2026-01-07T11:25:42.326610Z	info	proxy::inbound	listener established	address=[::]:15008 component="inbound" transparent=true
2026-01-07T11:25:42.326643Z	info	proxy::inbound_passthrough	listener established	address=[::]:15006 component="inbound plaintext" transparent=true
2026-01-07T11:25:42.326678Z	info	proxy::outbound	listener established	address=[::]:15001 component="outbound" transparent=true
2026-01-07T11:25:42.330073Z	info	inpod::statemanager	pod received, starting proxy	uid="53d03bc8-d94e-4780-9993-1da8dbaf372f" name="geneva-logger-mdm-hunting-detections-656c5ff565-tw2bl" namespace="wdatp-infra-system-logging"
2026-01-07T11:25:42.330300Z	info	dns::server	starting local DNS server	address=localhost:15053 component="dns"
2026-01-07T11:25:42.330433Z	error	inpod::workloadmanager	failed to start proxy: Bind([::1]:15053, Os { code: 99, kind: AddrNotAvailable, message: "Cannot assign requested address" })	uid=53d03bc8-d94e-4780-9993-1da8dbaf372f
2026-01-07T11:25:42.330664Z	error	access	connection complete	src.addr=10.8.8.130:51644 src.workload="geneva-logger-tvm-7d84d85d66-zvdr6" src.namespace="wdatp-infra-system-logging" src.identity="spiffe://cluster.local/ns/wdatp-infra-system-logging/sa/geneva-logger-op" dst.addr=10.8.1.168:15008 dst.hbone_addr=10.8.1.168:5001 dst.service="geneva-logger-mdsd-tvm.wdatp-infra-system-logging.svc.cluster.local" dst.workload="geneva-logger-mdsd-tvm-784fcc8db-wp2wb" dst.namespace="wdatp-infra-system-logging" dst.identity="spiffe://cluster.local/ns/wdatp-infra-system-logging/sa/geneva-logger-op" direction="outbound" bytes_sent=0 bytes_recv=0 duration="4ms" error="io error: Connection refused (os error 111)"
2026-01-07T11:25:42.336255Z	info	inpod::statemanager	pod received, starting proxy	uid="f05eb4da-264f-4430-a74a-0f3da1e7ad56" name="geneva-logger-mdsd-tvm-784fcc8db-vcw5n" namespace="wdatp-infra-system-logging"
2026-01-07T11:25:42.336594Z	info	dns::server	starting local DNS server	address=localhost:15053 component="dns"
2026-01-07T11:25:42.336758Z	info	proxy::inbound	listener established	address=[::]:15008 component="inbound" transparent=true
2026-01-07T11:25:42.336802Z	info	proxy::inbound_passthrough	listener established	address=[::]:15006 component="inbound plaintext" transparent=true
2026-01-07T11:25:42.336843Z	info	proxy::outbound	listener established	address=[::]:15001 component="outbound" transparent=true
2026-01-07T11:25:42.343482Z	info	inpod::statemanager	pod received, starting proxy	uid="2df5e2ba-db8d-4064-93a5-19c1bef813bd" name="geneva-logger-mdsd-md-mobile-5d96bcc696-cp82p" namespace="wdatp-infra-system-logging"
2026-01-07T11:25:42.343814Z	info	dns::server	starting local DNS server	address=localhost:15053 component="dns"
2026-01-07T11:25:42.343944Z	info	proxy::inbound	listener established	address=[::]:15008 component="inbound" transparent=true
2026-01-07T11:25:42.343975Z	info	proxy::inbound_passthrough	listener established	address=[::]:15006 component="inbound plaintext" transparent=true
2026-01-07T11:25:42.343998Z	info	proxy::outbound	listener established	address=[::]:15001 component="outbound" transparent=true
2026-01-07T11:25:42.349627Z	info	inpod::statemanager	pod received, starting proxy	uid="60cda57f-9377-4c54-ada5-f6081a95f8ca" name="geneva-logger-mdsd-sense-6b5cb94df6-lsb4j" namespace="wdatp-infra-system-logging"
2026-01-07T11:25:42.349900Z	info	dns::server	starting local DNS server	address=localhost:15053 component="dns"
2026-01-07T11:25:42.350076Z	info	proxy::inbound	listener established	address=[::]:15008 component="inbound" transparent=true
2026-01-07T11:25:42.350118Z	info	proxy::inbound_passthrough	listener established	address=[::]:15006 component="inbound plaintext" transparent=true
2026-01-07T11:25:42.350143Z	info	proxy::outbound	listener established	address=[::]:15001 component="outbound" transparent=true
2026-01-07T11:25:42.356723Z	info	inpod::statemanager	pod received, starting proxy	uid="6d2f6095-2271-4206-acfe-5ff6d9c0b29d" name="geneva-logger-band-64bb8b4b8-nkhgg" namespace="wdatp-infra-system-logging"
2026-01-07T11:25:42.356958Z	info	dns::server	starting local DNS server	address=localhost:15053 component="dns"
2026-01-07T11:25:42.357116Z	info	proxy::inbound	listener established	address=[::]:15008 component="inbound" transparent=true
2026-01-07T11:25:42.357148Z	info	proxy::inbound_passthrough	listener established	address=[::]:15006 component="inbound plaintext" transparent=true
2026-01-07T11:25:42.357180Z	info	proxy::outbound	listener established	address=[::]:15001 component="outbound" transparent=true
2026-01-07T11:25:42.361377Z	info	inpod::statemanager	pod received, starting proxy	uid="743b7f2d-243e-415b-b996-593f244af9cf" name="geneva-logger-tvm-7d84d85d66-6h6rl" namespace="wdatp-infra-system-logging"
2026-01-07T11:25:42.361631Z	info	dns::server	starting local DNS server	address=localhost:15053 component="dns"
2026-01-07T11:25:42.361790Z	info	proxy::inbound	listener established	address=[::]:15008 component="inbound" transparent=true
2026-01-07T11:25:42.361825Z	info	proxy::inbound_passthrough	listener established	address=[::]:15006 component="inbound plaintext" transparent=true
2026-01-07T11:25:42.361860Z	info	proxy::outbound	listener established	address=[::]:15001 component="outbound" transparent=true
2026-01-07T11:25:42.365397Z	info	inpod::statemanager	pod received, starting proxy	uid="9d5edb34-5e30-444e-a76a-5d2ce789b6e3" name="geneva-logger-ndr-5c7c56ddfc-gvz4v" namespace="wdatp-infra-system-logging"
2026-01-07T11:25:42.365671Z	info	dns::server	starting local DNS server	address=localhost:15053 component="dns"
2026-01-07T11:25:42.365849Z	info	proxy::inbound	listener established	address=[::]:15008 component="inbound" transparent=true
2026-01-07T11:25:42.365897Z	info	proxy::inbound_passthrough	listener established	address=[::]:15006 component="inbound plaintext" transparent=true
2026-01-07T11:25:42.365931Z	info	proxy::outbound	listener established	address=[::]:15001 component="outbound" transparent=true
2026-01-07T11:25:42.368661Z	info	inpod::statemanager	pod received, starting proxy	uid="53cb90e3-3f8c-4f39-bb88-fe0074faecf7" name="geneva-logger-mdsd-ine-audit-d95ff9d68-65m5r" namespace="wdatp-infra-system-logging"
2026-01-07T11:25:42.368896Z	info	dns::server	starting local DNS server	address=localhost:15053 component="dns"
2026-01-07T11:25:42.369064Z	info	proxy::inbound	listener established	address=[::]:15008 component="inbound" transparent=true
2026-01-07T11:25:42.369097Z	info	proxy::inbound_passthrough	listener established	address=[::]:15006 component="inbound plaintext" transparent=true
2026-01-07T11:25:42.369120Z	info	proxy::outbound	listener established	address=[::]:15001 component="outbound" transparent=true
2026-01-07T11:25:42.369873Z	info	inpod::statemanager	pod received, starting proxy	uid="7730af8c-2156-46b3-a7b2-321b5933523e" name="geneva-logger-mdm-m365dsre-bcfcc7f4-9h6hl" namespace="wdatp-infra-system-logging"
2026-01-07T11:25:42.370163Z	info	dns::server	starting local DNS server	address=localhost:15053 component="dns"
2026-01-07T11:25:42.370313Z	info	proxy::inbound	listener established	address=[::]:15008 component="inbound" transparent=true
2026-01-07T11:25:42.370352Z	info	proxy::inbound_passthrough	listener established	address=[::]:15006 component="inbound plaintext" transparent=true
2026-01-07T11:25:42.370380Z	info	proxy::outbound	listener established	address=[::]:15001 component="outbound" transparent=true
2026-01-07T11:25:42.370766Z	info	inpod::statemanager	received snapshot sent	
2026-01-07T11:25:42.521272Z	error	access	connection complete	src.addr=10.8.0.17:54274 src.workload="geneva-logger-band-64bb8b4b8-nkhgg" src.namespace="wdatp-infra-system-logging" src.identity="spiffe://cluster.local/ns/wdatp-infra-system-logging/sa/geneva-logger-op" dst.addr=10.8.5.193:15008 dst.hbone_addr=10.8.5.193:5001 dst.service="geneva-logger-mdsd-band.wdatp-infra-system-logging.svc.cluster.local" dst.workload="geneva-logger-mdsd-band-6bb4fb9647-mmbxr" dst.namespace="wdatp-infra-system-logging" dst.identity="spiffe://cluster.local/ns/wdatp-infra-system-logging/sa/geneva-logger-op" direction="outbound" bytes_sent=0 bytes_recv=0 duration="10ms" error="io error: Connection refused (os error 111)"
2026-01-07T11:25:42.602795Z	error	access	connection complete	src.addr=10.8.3.201:39154 src.workload="geneva-logger-tvm-7d84d85d66-j4vfc" src.namespace="wdatp-infra-system-logging" src.identity="spiffe://cluster.local/ns/wdatp-infra-system-logging/sa/geneva-logger-op" dst.addr=10.8.6.154:15008 dst.hbone_addr=10.8.6.154:5001 dst.service="geneva-logger-mdsd-tvm.wdatp-infra-system-logging.svc.cluster.local" dst.workload="geneva-logger-mdsd-tvm-784fcc8db-9rtzm" dst.namespace="wdatp-infra-system-logging" dst.identity="spiffe://cluster.local/ns/wdatp-infra-system-logging/sa/geneva-logger-op" direction="outbound" bytes_sent=0 bytes_recv=0 duration="2ms" error="io error: Connection refused (os error 111)"
2026-01-07T11:25:42.614723Z	error	access	connection complete	src.addr=10.8.9.38:39242 src.workload="geneva-logger-m365dsre-56765d6d58-gvw2j" src.namespace="wdatp-infra-system-logging" src.identity="spiffe://cluster.local/ns/wdatp-infra-system-logging/sa/geneva-logger-op" dst.addr=10.8.9.198:15008 dst.hbone_addr=10.8.9.198:5001 dst.service="geneva-logger-mdsd-m365dsre.wdatp-infra-system-logging.svc.cluster.local" dst.workload="geneva-logger-mdsd-m365dsre-79545f787-gf88k" dst.namespace="wdatp-infra-system-logging" dst.identity="spiffe://cluster.local/ns/wdatp-infra-system-logging/sa/geneva-logger-op" direction="outbound" bytes_sent=0 bytes_recv=0 duration="1ms" error="io error: Connection refused (os error 111)"
2026-01-07T11:25:42.723460Z	error	access	connection complete	src.addr=10.8.3.80:39618 src.workload="geneva-logger-sense-8657c59d6c-544fq" src.namespace="wdatp-infra-system-logging" src.identity="spiffe://cluster.local/ns/wdatp-infra-system-logging/sa/geneva-logger-op" dst.addr=10.8.10.69:15008 dst.hbone_addr=10.8.10.69:5001 dst.service="geneva-logger-mdsd-sense.wdatp-infra-system-logging.svc.cluster.local" dst.workload="geneva-logger-mdsd-sense-6b5cb94df6-kffx4" dst.namespace="wdatp-infra-system-logging" dst.identity="spiffe://cluster.local/ns/wdatp-infra-system-logging/sa/geneva-logger-op" direction="outbound" bytes_sent=0 bytes_recv=0 duration="1ms" error="io error: Connection refused (os error 111)"
2026-01-07T11:25:42.724689Z	info	xds::client:xds{id=1}	received response	type_url="type.googleapis.com/istio.workload.Address" size=1 removes=0
2026-01-07T11:25:42.938985Z	error	access	connection complete	src.addr=10.8.5.85:47914 src.workload="geneva-logger-wdatp-infra-system-logging-5686b8d89b-jpqj6" src.namespace="wdatp-infra-system-logging" src.identity="spiffe://cluster.local/ns/wdatp-infra-system-logging/sa/geneva-logger-op" dst.addr=10.8.10.50:15008 dst.hbone_addr=10.8.10.50:5001 dst.service="geneva-logger-mdsd-wdatp-infra-system-logging.wdatp-infra-system-logging.svc.cluster.local" dst.workload="geneva-logger-mdsd-wdatp-infra-system-logging-c47f74c59-mmtzl" dst.namespace="wdatp-infra-system-logging" dst.identity="spiffe://cluster.local/ns/wdatp-infra-system-logging/sa/geneva-logger-op" direction="outbound" bytes_sent=0 bytes_recv=0 duration="0ms" error="io error: Connection refused (os error 111)"
2026-01-07T11:25:42.958614Z	error	access	connection complete	src.addr=10.8.4.78:38724 src.workload="geneva-logger-mde-disruption-5494985645-d8bvh" src.namespace="wdatp-infra-system-logging" src.identity="spiffe://cluster.local/ns/wdatp-infra-system-logging/sa/geneva-logger-op" dst.addr=10.8.6.173:15008 dst.hbone_addr=10.8.6.173:5001 dst.service="geneva-logger-mdsd-mde-disruption.wdatp-infra-system-logging.svc.cluster.local" dst.workload="geneva-logger-mdsd-mde-disruption-6b8c495b99-b45hq" dst.namespace="wdatp-infra-system-logging" dst.identity="spiffe://cluster.local/ns/wdatp-infra-system-logging/sa/geneva-logger-op" direction="outbound" bytes_sent=0 bytes_recv=0 duration="2ms" error="io error: Connection refused (os error 111)"
2026-01-07T11:25:43.034695Z	error	access	connection complete	src.addr=10.8.8.230:58132 src.workload="geneva-logger-ndr-5c7c56ddfc-gvz4v" src.namespace="wdatp-infra-system-logging" src.identity="spiffe://cluster.local/ns/wdatp-infra-system-logging/sa/geneva-logger-op" dst.addr=10.8.3.82:15008 dst.hbone_addr=10.8.3.82:5001 dst.service="geneva-logger-mdsd-ndr.wdatp-infra-system-logging.svc.cluster.local" dst.workload="geneva-logger-mdsd-ndr-847cd599d6-bxcck" dst.namespace="wdatp-infra-system-logging" dst.identity="spiffe://cluster.local/ns/wdatp-infra-system-logging/sa/geneva-logger-op" direction="outbound" bytes_sent=0 bytes_recv=0 duration="12ms" error="io error: Connection refused (os error 111)"
2026-01-07T11:25:43.220964Z	error	access	connection complete	src.addr=10.8.7.10:51960 src.workload="geneva-logger-tvm-7d84d85d66-6h6rl" src.namespace="wdatp-infra-system-logging" src.identity="spiffe://cluster.local/ns/wdatp-infra-system-logging/sa/geneva-logger-op" dst.addr=10.8.6.154:15008 dst.hbone_addr=10.8.6.154:5001 dst.service="geneva-logger-mdsd-tvm.wdatp-infra-system-logging.svc.cluster.local" dst.workload="geneva-logger-mdsd-tvm-784fcc8db-9rtzm" dst.namespace="wdatp-infra-system-logging" dst.identity="spiffe://cluster.local/ns/wdatp-infra-system-logging/sa/geneva-logger-op" direction="outbound" bytes_sent=0 bytes_recv=0 duration="2ms" error="io error: Connection refused (os error 111)"
2026-01-07T11:25:43.308464Z	info	xds::client:xds{id=1}	received response	type_url="type.googleapis.com/istio.workload.Address" size=2 removes=0
2026-01-07T11:25:43.399888Z	error	access	connection complete	src.addr=10.8.8.130:51660 src.workload="geneva-logger-tvm-7d84d85d66-zvdr6" src.namespace="wdatp-infra-system-logging" src.identity="spiffe://cluster.local/ns/wdatp-infra-system-logging/sa/geneva-logger-op" dst.addr=10.8.0.191:15008 dst.hbone_addr=10.8.0.191:5001 dst.service="geneva-logger-mdsd-tvm.wdatp-infra-system-logging.svc.cluster.local" dst.workload="geneva-logger-mdsd-tvm-784fcc8db-pdh9m" dst.namespace="wdatp-infra-system-logging" dst.identity="spiffe://cluster.local/ns/wdatp-infra-system-logging/sa/geneva-logger-op" direction="outbound" bytes_sent=0 bytes_recv=0 duration="2ms" error="io error: Connection refused (os error 111)"
2026-01-07T11:25:43.576008Z	info	inpod::statemanager	pod received, starting proxy	uid="0fa94b23-7d27-40ad-b8b3-7e472bb189e7" name="geneva-logger-tvm-7d84d85d66-qc8d2" namespace="wdatp-infra-system-logging"
2026-01-07T11:25:43.576276Z	info	dns::server	starting local DNS server	address=localhost:15053 component="dns"
2026-01-07T11:25:43.576521Z	info	proxy::inbound	listener established	address=[::]:15008 component="inbound" transparent=true
2026-01-07T11:25:43.576565Z	info	proxy::inbound_passthrough	listener established	address=[::]:15006 component="inbound plaintext" transparent=true
2026-01-07T11:25:43.576590Z	info	proxy::outbound	listener established	address=[::]:15001 component="outbound" transparent=true
2026-01-07T11:25:43.732104Z	info	access	connection complete	src.addr=10.8.3.80:47444 src.workload="geneva-logger-sense-8657c59d6c-544fq" src.namespace="wdatp-infra-system-logging" src.identity="spiffe://cluster.local/ns/wdatp-infra-system-logging/sa/geneva-logger-op" dst.addr=10.8.0.79:15008 dst.hbone_addr=10.8.0.79:5001 dst.service="geneva-logger-mdsd-sense.wdatp-infra-system-logging.svc.cluster.local" dst.workload="geneva-logger-mdsd-sense-6b5cb94df6-lsb4j" dst.namespace="wdatp-infra-system-logging" dst.identity="spiffe://cluster.local/ns/wdatp-infra-system-logging/sa/geneva-logger-op" direction="inbound" bytes_sent=0 bytes_recv=0 duration="1ms"
2026-01-07T11:25:43.733051Z	info	access	connection complete	src.addr=10.8.3.80:39628 src.workload="geneva-logger-sense-8657c59d6c-544fq" src.namespace="wdatp-infra-system-logging" src.identity="spiffe://cluster.local/ns/wdatp-infra-system-logging/sa/geneva-logger-op" dst.addr=10.8.0.79:15008 dst.hbone_addr=10.8.0.79:5001 dst.service="geneva-logger-mdsd-sense.wdatp-infra-system-logging.svc.cluster.local" dst.workload="geneva-logger-mdsd-sense-6b5cb94df6-lsb4j" dst.namespace="wdatp-infra-system-logging" dst.identity="spiffe://cluster.local/ns/wdatp-infra-system-logging/sa/geneva-logger-op" direction="outbound" bytes_sent=0 bytes_recv=0 duration="10ms"
2026-01-07T11:25:43.775185Z	info	access	connection complete	src.addr=10.8.3.201:39168 src.workload="geneva-logger-tvm-7d84d85d66-j4vfc" src.namespace="wdatp-infra-system-logging" src.identity="spiffe://cluster.local/ns/wdatp-infra-system-logging/sa/geneva-logger-op" dst.addr=10.8.19.88:15008 dst.hbone_addr=10.8.19.88:5001 dst.service="geneva-logger-mdsd-tvm.wdatp-infra-system-logging.svc.cluster.local" dst.workload="geneva-logger-mdsd-tvm-784fcc8db-bbdk5" dst.namespace="wdatp-infra-system-logging" dst.identity="spiffe://cluster.local/ns/wdatp-infra-system-logging/sa/geneva-logger-op" direction="outbound" bytes_sent=0 bytes_recv=0 duration="66ms"
2026-01-07T11:25:43.834089Z	info	xds::client:xds{id=1}	received response	type_url="type.googleapis.com/istio.workload.Address" size=1 removes=0
2026-01-07T11:25:44.029933Z	info	xds::client:xds{id=1}	received response	type_url="type.googleapis.com/istio.workload.Address" size=1 removes=0
2026-01-07T11:25:44.046905Z	error	access	connection complete	src.addr=10.8.4.78:38726 src.workload="geneva-logger-mde-disruption-5494985645-d8bvh" src.namespace="wdatp-infra-system-logging" src.identity="spiffe://cluster.local/ns/wdatp-infra-system-logging/sa/geneva-logger-op" dst.addr=10.8.5.160:15008 dst.hbone_addr=10.8.5.160:5001 dst.service="geneva-logger-mdsd-mde-disruption.wdatp-infra-system-logging.svc.cluster.local" dst.workload="geneva-logger-mdsd-mde-disruption-6b8c495b99-vw2p8" dst.namespace="wdatp-infra-system-logging" dst.identity="spiffe://cluster.local/ns/wdatp-infra-system-logging/sa/geneva-logger-op" direction="outbound" bytes_sent=0 bytes_recv=0 duration="2ms" error="io error: Connection refused (os error 111)"
2026-01-07T11:25:44.243137Z	error	access	connection complete	src.addr=10.8.7.10:51976 src.workload="geneva-logger-tvm-7d84d85d66-6h6rl" src.namespace="wdatp-infra-system-logging" src.identity="spiffe://cluster.local/ns/wdatp-infra-system-logging/sa/geneva-logger-op" dst.addr=10.8.10.73:15008 dst.hbone_addr=10.8.10.73:5001 dst.service="geneva-logger-mdsd-tvm.wdatp-infra-system-logging.svc.cluster.local" dst.workload="geneva-logger-mdsd-tvm-784fcc8db-rs69t" dst.namespace="wdatp-infra-system-logging" dst.identity="spiffe://cluster.local/ns/wdatp-infra-system-logging/sa/geneva-logger-op" direction="outbound" bytes_sent=0 bytes_recv=0 duration="1ms" error="io error: Connection refused (os error 111)"
2026-01-07T11:25:44.261916Z	info	xds::client:xds{id=1}	received response	type_url="type.googleapis.com/istio.workload.Address" size=2 removes=0
2026-01-07T11:25:44.320211Z	error	access	connection complete	src.addr=10.8.8.130:51676 src.workload="geneva-logger-tvm-7d84d85d66-zvdr6" src.namespace="wdatp-infra-system-logging" src.identity="spiffe://cluster.local/ns/wdatp-infra-system-logging/sa/geneva-logger-op" dst.addr=10.8.5.214:15008 dst.hbone_addr=10.8.5.214:5001 dst.service="geneva-logger-mdsd-tvm.wdatp-infra-system-logging.svc.cluster.local" dst.workload="geneva-logger-mdsd-tvm-784fcc8db-jnmn5" dst.namespace="wdatp-infra-system-logging" dst.identity="spiffe://cluster.local/ns/wdatp-infra-system-logging/sa/geneva-logger-op" direction="outbound" bytes_sent=0 bytes_recv=0 duration="2ms" error="io error: Connection refused (os error 111)"
2026-01-07T11:25:44.722441Z	info	access	connection complete	src.addr=10.8.3.80:47444 src.workload="geneva-logger-sense-8657c59d6c-544fq" src.namespace="wdatp-infra-system-logging" src.identity="spiffe://cluster.local/ns/wdatp-infra-system-logging/sa/geneva-logger-op" dst.addr=10.8.0.79:15008 dst.hbone_addr=10.8.0.79:5001 dst.service="geneva-logger-mdsd-sense.wdatp-infra-system-logging.svc.cluster.local" dst.workload="geneva-logger-mdsd-sense-6b5cb94df6-lsb4j" dst.namespace="wdatp-infra-system-logging" dst.identity="spiffe://cluster.local/ns/wdatp-infra-system-logging/sa/geneva-logger-op" direction="inbound" bytes_sent=0 bytes_recv=0 duration="0ms"
2026-01-07T11:25:44.722595Z	info	access	connection complete	src.addr=10.8.3.80:39632 src.workload="geneva-logger-sense-8657c59d6c-544fq" src.namespace="wdatp-infra-system-logging" src.identity="spiffe://cluster.local/ns/wdatp-infra-system-logging/sa/geneva-logger-op" dst.addr=10.8.0.79:15008 dst.hbone_addr=10.8.0.79:5001 dst.service="geneva-logger-mdsd-sense.wdatp-infra-system-logging.svc.cluster.local" dst.workload="geneva-logger-mdsd-sense-6b5cb94df6-lsb4j" dst.namespace="wdatp-infra-system-logging" dst.identity="spiffe://cluster.local/ns/wdatp-infra-system-logging/sa/geneva-logger-op" direction="outbound" bytes_sent=0 bytes_recv=0 duration="0ms"
2026-01-07T11:25:44.852996Z	error	access	connection complete	src.addr=10.8.17.73:49208 src.workload="prometheus-infra-monitoring-prometheu-prometheus-0" src.namespace="wdatp-infra-system" dst.addr=10.8.7.72:24231 dst.service="geneva-logger-kube-system.wdatp-infra-system-logging.svc.cluster.local" dst.workload="geneva-logger-kube-system-5856ccd854-79frs" dst.namespace="wdatp-infra-system-logging" direction="inbound" bytes_sent=0 bytes_recv=0 duration="0ms" error="connection failed: Connection refused (os error 111)"
2026-01-07T11:25:45.028463Z	error	access	connection complete	src.addr=10.8.4.78:38728 src.workload="geneva-logger-mde-disruption-5494985645-d8bvh" src.namespace="wdatp-infra-system-logging" src.identity="spiffe://cluster.local/ns/wdatp-infra-system-logging/sa/geneva-logger-op" dst.addr=10.8.6.173:15008 dst.hbone_addr=10.8.6.173:5001 dst.service="geneva-logger-mdsd-mde-disruption.wdatp-infra-system-logging.svc.cluster.local" dst.workload="geneva-logger-mdsd-mde-disruption-6b8c495b99-b45hq" dst.namespace="wdatp-infra-system-logging" dst.identity="spiffe://cluster.local/ns/wdatp-infra-system-logging/sa/geneva-logger-op" direction="outbound" bytes_sent=0 bytes_recv=0 duration="1ms" error="io error: Connection refused (os error 111)"
2026-01-07T11:25:45.159533Z	info	access	connection complete	src.addr=10.8.3.201:58168 src.workload="geneva-logger-tvm-7d84d85d66-j4vfc" src.namespace="wdatp-infra-system-logging" src.identity="spiffe://cluster.local/ns/wdatp-infra-system-logging/sa/geneva-logger-op" dst.addr=10.8.0.148:15008 dst.hbone_addr=10.8.0.148:5001 dst.service="geneva-logger-mdsd-tvm.wdatp-infra-system-logging.svc.cluster.local" dst.workload="geneva-logger-mdsd-tvm-784fcc8db-vcw5n" dst.namespace="wdatp-infra-system-logging" dst.identity="spiffe://cluster.local/ns/wdatp-infra-system-logging/sa/geneva-logger-op" direction="inbound" bytes_sent=0 bytes_recv=0 duration="545ms"
2026-01-07T11:25:45.159719Z	info	access	connection complete	src.addr=10.8.3.201:39170 src.workload="geneva-logger-tvm-7d84d85d66-j4vfc" src.namespace="wdatp-infra-system-logging" src.identity="spiffe://cluster.local/ns/wdatp-infra-system-logging/sa/geneva-logger-op" dst.addr=10.8.0.148:15008 dst.hbone_addr=10.8.0.148:5001 dst.service="geneva-logger-mdsd-tvm.wdatp-infra-system-logging.svc.cluster.local" dst.workload="geneva-logger-mdsd-tvm-784fcc8db-vcw5n" dst.namespace="wdatp-infra-system-logging" dst.identity="spiffe://cluster.local/ns/wdatp-infra-system-logging/sa/geneva-logger-op" direction="outbound" bytes_sent=0 bytes_recv=0 duration="552ms"
2026-01-07T11:25:45.190118Z	info	xds::client:xds{id=1}	received response	type_url="type.googleapis.com/istio.workload.Address" size=1 removes=0
2026-01-07T11:25:45.265587Z	error	access	connection complete	src.addr=10.8.7.10:51068 src.workload="geneva-logger-tvm-7d84d85d66-6h6rl" src.namespace="wdatp-infra-system-logging" src.identity="spiffe://cluster.local/ns/wdatp-infra-system-logging/sa/geneva-logger-op" dst.addr=10.8.5.235:15008 dst.hbone_addr=10.8.5.235:5001 dst.service="geneva-logger-mdsd-tvm.wdatp-infra-system-logging.svc.cluster.local" dst.workload="geneva-logger-mdsd-tvm-784fcc8db-kdt26" dst.namespace="wdatp-infra-system-logging" dst.identity="spiffe://cluster.local/ns/wdatp-infra-system-logging/sa/geneva-logger-op" direction="outbound" bytes_sent=0 bytes_recv=0 duration="10ms" error="io error: Connection refused (os error 111)"
2026-01-07T11:25:45.354377Z	error	access	connection complete	src.addr=10.8.8.130:46774 src.workload="geneva-logger-tvm-7d84d85d66-zvdr6" src.namespace="wdatp-infra-system-logging" src.identity="spiffe://cluster.local/ns/wdatp-infra-system-logging/sa/geneva-logger-op" dst.addr=10.8.5.225:15008 dst.hbone_addr=10.8.5.225:5001 dst.service="geneva-logger-mdsd-tvm.wdatp-infra-system-logging.svc.cluster.local" dst.workload="geneva-logger-mdsd-tvm-784fcc8db-x976v" dst.namespace="wdatp-infra-system-logging" dst.identity="spiffe://cluster.local/ns/wdatp-infra-system-logging/sa/geneva-logger-op" direction="outbound" bytes_sent=0 bytes_recv=0 duration="2ms" error="io error: Connection refused (os error 111)"
2026-01-07T11:25:45.536293Z	info	xds::client:xds{id=1}	received response	type_url="type.googleapis.com/istio.workload.Address" size=1 removes=0
2026-01-07T11:25:45.619440Z	error	access	connection complete	src.addr=10.8.3.201:47754 src.workload="geneva-logger-tvm-7d84d85d66-j4vfc" src.namespace="wdatp-infra-system-logging" src.identity="spiffe://cluster.local/ns/wdatp-infra-system-logging/sa/geneva-logger-op" dst.addr=10.8.6.154:15008 dst.hbone_addr=10.8.6.154:5001 dst.service="geneva-logger-mdsd-tvm.wdatp-infra-system-logging.svc.cluster.local" dst.workload="geneva-logger-mdsd-tvm-784fcc8db-9rtzm" dst.namespace="wdatp-infra-system-logging" dst.identity="spiffe://cluster.local/ns/wdatp-infra-system-logging/sa/geneva-logger-op" direction="outbound" bytes_sent=0 bytes_recv=0 duration="2ms" error="io error: Connection refused (os error 111)"
2026-01-07T11:25:45.723233Z	error	access	connection complete	src.addr=10.8.3.80:39412 src.workload="geneva-logger-sense-8657c59d6c-544fq" src.namespace="wdatp-infra-system-logging" src.identity="spiffe://cluster.local/ns/wdatp-infra-system-logging/sa/geneva-logger-op" dst.addr=10.8.1.153:15008 dst.hbone_addr=10.8.1.153:5001 dst.service="geneva-logger-mdsd-sense.wdatp-infra-system-logging.svc.cluster.local" dst.workload="geneva-logger-mdsd-sense-6b5cb94df6-blxlc" dst.namespace="wdatp-infra-system-logging" dst.identity="spiffe://cluster.local/ns/wdatp-infra-system-logging/sa/geneva-logger-op" direction="outbound" bytes_sent=0 bytes_recv=0 duration="1ms" error="io error: Connection refused (os error 111)"
2026-01-07T11:25:46.047430Z	error	access	connection complete	src.addr=10.8.4.78:37212 src.workload="geneva-logger-mde-disruption-5494985645-d8bvh" src.namespace="wdatp-infra-system-logging" src.identity="spiffe://cluster.local/ns/wdatp-infra-system-logging/sa/geneva-logger-op" dst.addr=10.8.6.173:15008 dst.hbone_addr=10.8.6.173:5001 dst.service="geneva-logger-mdsd-mde-disruption.wdatp-infra-system-logging.svc.cluster.local" dst.workload="geneva-logger-mdsd-mde-disruption-6b8c495b99-b45hq" dst.namespace="wdatp-infra-system-logging" dst.identity="spiffe://cluster.local/ns/wdatp-infra-system-logging/sa/geneva-logger-op" direction="outbound" bytes_sent=0 bytes_recv=0 duration="1ms" error="io error: Connection refused (os error 111)"
2026-01-07T11:25:46.226127Z	error	access	connection complete	src.addr=10.8.7.10:51074 src.workload="geneva-logger-tvm-7d84d85d66-6h6rl" src.namespace="wdatp-infra-system-logging" src.identity="spiffe://cluster.local/ns/wdatp-infra-system-logging/sa/geneva-logger-op" dst.addr=10.8.6.151:15008 dst.hbone_addr=10.8.6.151:5001 dst.service="geneva-logger-mdsd-tvm.wdatp-infra-system-logging.svc.cluster.local" dst.workload="geneva-logger-mdsd-tvm-784fcc8db-j7xg2" dst.namespace="wdatp-infra-system-logging" dst.identity="spiffe://cluster.local/ns/wdatp-infra-system-logging/sa/geneva-logger-op" direction="outbound" bytes_sent=0 bytes_recv=0 duration="2ms" error="io error: Connection refused (os error 111)"
2026-01-07T11:25:46.278355Z	info	xds::client:xds{id=1}	received response	type_url="type.googleapis.com/istio.workload.Address" size=1 removes=0
2026-01-07T11:25:46.316753Z	error	access	connection complete	src.addr=10.8.8.130:46788 src.workload="geneva-logger-tvm-7d84d85d66-zvdr6" src.namespace="wdatp-infra-system-logging" src.identity="spiffe://cluster.local/ns/wdatp-infra-system-logging/sa/geneva-logger-op" dst.addr=10.8.6.151:15008 dst.hbone_addr=10.8.6.151:5001 dst.service="geneva-logger-mdsd-tvm.wdatp-infra-system-logging.svc.cluster.local" dst.workload="geneva-logger-mdsd-tvm-784fcc8db-j7xg2" dst.namespace="wdatp-infra-system-logging" dst.identity="spiffe://cluster.local/ns/wdatp-infra-system-logging/sa/geneva-logger-op" direction="outbound" bytes_sent=0 bytes_recv=0 duration="3ms" error="io error: Connection refused (os error 111)"
2026-01-07T11:25:46.524143Z	info	xds::client:xds{id=1}	received response	type_url="type.googleapis.com/istio.workload.Address" size=1 removes=0
2026-01-07T11:25:46.652262Z	info	access	connection complete	src.addr=10.8.3.201:47768 src.workload="geneva-logger-tvm-7d84d85d66-j4vfc" src.namespace="wdatp-infra-system-logging" src.identity="spiffe://cluster.local/ns/wdatp-infra-system-logging/sa/geneva-logger-op" dst.addr=10.8.19.88:15008 dst.hbone_addr=10.8.19.88:5001 dst.service="geneva-logger-mdsd-tvm.wdatp-infra-system-logging.svc.cluster.local" dst.workload="geneva-logger-mdsd-tvm-784fcc8db-bbdk5" dst.namespace="wdatp-infra-system-logging" dst.identity="spiffe://cluster.local/ns/wdatp-infra-system-logging/sa/geneva-logger-op" direction="outbound" bytes_sent=0 bytes_recv=0 duration="3ms"
2026-01-07T11:25:46.799189Z	error	access	connection complete	src.addr=10.8.3.80:39416 src.workload="geneva-logger-sense-8657c59d6c-544fq" src.namespace="wdatp-infra-system-logging" src.identity="spiffe://cluster.local/ns/wdatp-infra-system-logging/sa/geneva-logger-op" dst.addr=10.8.9.74:15008 dst.hbone_addr=10.8.9.74:5001 dst.service="geneva-logger-mdsd-sense.wdatp-infra-system-logging.svc.cluster.local" dst.workload="geneva-logger-mdsd-sense-6b5cb94df6-phfg9" dst.namespace="wdatp-infra-system-logging" dst.identity="spiffe://cluster.local/ns/wdatp-infra-system-logging/sa/geneva-logger-op" direction="outbound" bytes_sent=0 bytes_recv=0 duration="1ms" error="io error: Connection refused (os error 111)"
2026-01-07T11:25:47.051219Z	error	access	connection complete	src.addr=10.8.4.78:37214 src.workload="geneva-logger-mde-disruption-5494985645-d8bvh" src.namespace="wdatp-infra-system-logging" src.identity="spiffe://cluster.local/ns/wdatp-infra-system-logging/sa/geneva-logger-op" dst.addr=10.8.5.160:15008 dst.hbone_addr=10.8.5.160:5001 dst.service="geneva-logger-mdsd-mde-disruption.wdatp-infra-system-logging.svc.cluster.local" dst.workload="geneva-logger-mdsd-mde-disruption-6b8c495b99-vw2p8" dst.namespace="wdatp-infra-system-logging" dst.identity="spiffe://cluster.local/ns/wdatp-infra-system-logging/sa/geneva-logger-op" direction="outbound" bytes_sent=0 bytes_recv=0 duration="2ms" error="io error: Connection refused (os error 111)"
2026-01-07T11:25:47.166647Z	info	inpod::statemanager	retrying workload	uid="53d03bc8-d94e-4780-9993-1da8dbaf372f"
2026-01-07T11:25:47.166940Z	info	dns::server	starting local DNS server	address=localhost:15053 component="dns"
2026-01-07T11:25:47.167097Z	info	inpod::statemanager	retrying workload failed: failed to bind to address [::1]:15053: Cannot assign requested address (os error 99)	uid="53d03bc8-d94e-4780-9993-1da8dbaf372f"
2026-01-07T11:25:47.167118Z	info	inpod::statemanager	retrying workload	uid="40019417-daea-40dc-9ce2-6fb8d3b27662"
2026-01-07T11:25:47.167325Z	info	dns::server	starting local DNS server	address=localhost:15053 component="dns"
2026-01-07T11:25:47.167457Z	info	inpod::statemanager	retrying workload failed: failed to bind to address [::1]:15053: Cannot assign requested address (os error 99)	uid="40019417-daea-40dc-9ce2-6fb8d3b27662"
2026-01-07T11:25:47.167471Z	info	inpod::workloadmanager	scheduling retry	uids=["40019417-daea-40dc-9ce2-6fb8d3b27662", "53d03bc8-d94e-4780-9993-1da8dbaf372f"]

2. ztunnel deployment:

apiVersion: v1
kind: Pod
metadata:
  annotations:
    prometheus.io/port: "15020"
    prometheus.io/scrape: "true"
    sidecar.istio.io/inject: "false"
  creationTimestamp: "2026-01-09T00:01:04Z"
  generateName: ztunnel-
  generation: 1
  labels:
    BuildID: "20260107.5"
    app: ztunnel
    app.kubernetes.io/instance: istio
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: ztunnel
    app.kubernetes.io/part-of: istio
    app.kubernetes.io/version: 1.28.0
    controller-revision-hash: 66776cbfcd
    helm.sh/chart: istioZtunnel-1.28.0
    istio.io/dataplane-mode: none
    pod-template-generation: "1"
    sidecar.istio.io/inject: "false"
  name: ztunnel-frx6d
  namespace: istio
  ownerReferences:
  - apiVersion: apps/v1
    blockOwnerDeletion: true
    controller: true
    kind: DaemonSet
    name: ztunnel
    uid: ac585a84-1151-448f-b704-dff917bbd89e
  resourceVersion: "56060437"
  uid: ad34ed0c-6950-451c-a15d-ba1cc1ca7098
spec:
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
        - matchFields:
          - key: metadata.name
            operator: In
            values:
            - aks-default-dk8s-system-pool-amd-dads-v5-2lvf4
  containers:
  - args:
    - proxy
    - ztunnel
    env:
    - name: CA_ADDRESS
      value: istiod.istio.svc:15012
    - name: XDS_ADDRESS
      value: istiod.istio.svc:15012
    - name: RUST_LOG
      value: info
    - name: RUST_BACKTRACE
      value: "1"
    - name: ISTIO_META_CLUSTER_ID
      value: Kubernetes
    - name: INPOD_ENABLED
      value: "true"
    - name: TERMINATION_GRACE_PERIOD_SECONDS
      value: "30"
    - name: POD_NAME
      valueFrom:
        fieldRef:
          apiVersion: v1
          fieldPath: metadata.name
    - name: POD_NAMESPACE
      valueFrom:
        fieldRef:
          apiVersion: v1
          fieldPath: metadata.namespace
    - name: NODE_NAME
      valueFrom:
        fieldRef:
          apiVersion: v1
          fieldPath: spec.nodeName
    - name: INSTANCE_IP
      valueFrom:
        fieldRef:
          apiVersion: v1
          fieldPath: status.podIP
    - name: SERVICE_ACCOUNT
      valueFrom:
        fieldRef:
          apiVersion: v1
          fieldPath: spec.serviceAccountName
    - name: ISTIO_META_ENABLE_HBONE
      value: "true"
    - name: ZTUNNEL_CPU_LIMIT
      valueFrom:
        resourceFieldRef:
          divisor: "0"
          resource: limits.cpu
    image: wdatpacreus.azurecr.io/ztunnel:v1.28.0-official-20260107.5
    imagePullPolicy: IfNotPresent
    name: istio-proxy
    ports:
    - containerPort: 15020
      name: ztunnel-stats
      protocol: TCP
    readinessProbe:
      failureThreshold: 3
      httpGet:
        path: /healthz/ready
        port: 15021
        scheme: HTTP
      periodSeconds: 10
      successThreshold: 1
      timeoutSeconds: 1
    resources:
      limits:
        memory: 1500Mi
      requests:
        cpu: 200m
        memory: 200Mi
    securityContext:
      allowPrivilegeEscalation: true
      capabilities:
        add:
        - NET_ADMIN
        - SYS_ADMIN
        - NET_RAW
        drop:
        - ALL
      privileged: false
      readOnlyRootFilesystem: true
      runAsGroup: 1337
      runAsNonRoot: false
      runAsUser: 0
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    volumeMounts:
    - mountPath: /var/run/secrets/istio
      name: istiod-ca-cert
    - mountPath: /var/run/secrets/tokens
      name: istio-token
    - mountPath: /var/run/ztunnel
      name: cni-ztunnel-sock-dir
    - mountPath: /tmp
      name: tmp
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: kube-api-access-kmnz2
      readOnly: true
  dnsPolicy: ClusterFirst
  enableServiceLinks: true

3. we have a large set of fleet. We didn't face this issue on all clusters. Recently we started on prod clusters and we reverted the changes to exclude those workload outside mesh. We didn't see the issue in stg clusters. But in prod, where we have huge set of traffic, we faced in 2-3 clusters.

[karan56625]

all cluster is having "ipFamilies": [ "IPv4"],

[Stevenjin8]

@karan56625 Are you doing anything with ipv6 loopback sysctl configuration? I'm guessing not. I have to asks.

1. Could you true setting the IPV6_ENABLED: "false" in your ztunnel deployment?
2. I want to try to replicate this issue on AKS. What are your cluster network settings? Which CNI are you using? Anything else you think could be relevant?

[karan56625]

1. sure. But I am still trying to replicate this on staging.
2. It was observed on both pod subnet and overlay.
   @Stevenjin8 , can we connect over slack to discuss this?

[Stevenjin8]

@karan56625 yes feel free to dm me. You can also email me at sjinxuan@microsoft.com if you feel more comfortable with that.