Healthcheck port lacks an entry causing Neuvector to fail when deployed in a service mesh

[daxmc99]

When Neuvector is run within an Istio service mesh, Istio requires that all interpod communication be defined in the application template.

For example,

neuvector-helm/charts/core/templates/controller-service.yaml

Lines 11 to 21 in 27da6e2

	clusterIP: None
	ports:
	- port: 18300
	protocol: "TCP"
	name: "cluster-tcp-18300"
	- port: 18301
	protocol: "TCP"
	name: "cluster-tcp-18301"
	- port: 18301
	protocol: "UDP"
	name: "cluster-udp-18301"

However, the upgrader
https://github.com/neuvector/neuvector/blob/0155dd77b28bdf8c15340f4f7f87f8c77e414322/upgrader/postsync.go#L212
depends on access to port 18500.

This causes the cert-upgrader job to continually fail and create BlackHole entries in the istio proxy sidecar.

BlackHole logs

## Logs

[pod/neuvector-cert-upgrader-job-s9mvl/istio-proxy] [2025-04-22T22:59:39.046Z] "- - -" 0 UH - - "-" 0 0 0 - "-" "-" "-" "-" "-" BlackHoleCluster - 172.21.0.43:18500 172.21.0.39:42836 - -
[pod/neuvector-cert-upgrader-job-s9mvl/istio-proxy] [2025-04-22T22:59:44.110Z] "- - -" 0 UH - - "-" 0 0 0 - "-" "-" "-" "-" "-" BlackHoleCluster - 172.21.0.43:18500 172.21.0.39:47832 - -
[pod/neuvector-cert-upgrader-job-s9mvl/istio-proxy] [2025-04-22T22:59:49.420Z] "- - -" 0 UH - - "-" 0 0 0 - "-" "-" "-" "-" "-" BlackHoleCluster - 172.21.0.43:18500 172.21.0.39:47836 - -
[pod/neuvector-cert-upgrader-job-s9mvl/istio-proxy] [2025-04-22T22:59:54.629Z] "- - -" 0 UH - - "-" 0 0 0 - "-" "-" "-" "-" "-" BlackHoleCluster - 172.21.0.43:18500 172.21.0.39:58298 - -
[pod/neuvector-cert-upgrader-job-s9mvl/istio-proxy] [2025-04-22T22:59:59.851Z] "- - -" 0 UH - - "-" 0 0 0 - "-" "-" "-" "-" "-" BlackHoleCluster - 172.21.0.43:18500 172.21.0.39:58308 - -
[pod/neuvector-cert-upgrader-job-s9mvl/istio-proxy] [2025-04-22T23:00:05.303Z] "- - -" 0 UH - - "-" 0 0 0 - "-" "-" "-" "-" "-" BlackHoleCluster - 172.21.0.43:18500 172.21.0.39:34026 - -
[pod/neuvector-cert-upgrader-job-s9mvl/istio-proxy] [2025-04-22T23:00:10.794Z] "- - -" 0 UH - - "-" 0 0 0 - "-" "-" "-" "-" "-" BlackHoleCluster - 172.21.0.43:18500 172.21.0.39:34042 - -
[pod/neuvector-cert-upgrader-job-s9mvl/istio-proxy] [2025-04-22T23:00:15.890Z] "- - -" 0 UH - - "-" 0 0 0 - "-" "-" "-" "-" "-" BlackHoleCluster - 172.21.0.43:18500 172.21.0.39:58650 - -
[pod/neuvector-cert-upgrader-job-s9mvl/istio-proxy] [2025-04-22T23:00:21.319Z] "- - -" 0 UH - - "-" 0 0 0 - "-" "-" "-" "-" "-" BlackHoleCluster - 172.21.0.43:18500 172.21.0.39:58664 - -
[pod/neuvector-cert-upgrader-job-s9mvl/istio-proxy] [2025-04-22T23:00:26.542Z] "- - -" 0 UH - - "-" 0 0 0 - "-" "-" "-" "-" "-" BlackHoleCluster - 172.21.0.43:18500 172.21.0.39:50412 - -
[pod/neuvector-cert-upgrader-job-s9mvl/istio-proxy] [2025-04-22T23:00:31.754Z] "- - -" 0 UH - - "-" 0 0 0 - "-" "-" "-" "-" "-" BlackHoleCluster - 172.21.0.43:18500 172.21.0.39:50420 - -
[pod/neuvector-cert-upgrader-job-s9mvl/istio-proxy] [2025-04-22T23:00:36.995Z] "- - -" 0 UH - - "-" 0 0 0 - "-" "-" "-" "-" "-" BlackHoleCluster - 172.21.0.43:18500 172.21.0.39:44944 - -
[pod/neuvector-cert-upgrader-job-s9mvl/istio-proxy] [2025-04-22T23:00:42.103Z] "- - -" 0 UH - - "-" 0 0 0 - "-" "-" "-" "-" "-" BlackHoleCluster - 172.21.0.43:18500 172.21.0.39:44960 - -
[pod/neuvector-cert-upgrader-job-s9mvl/istio-proxy] [2025-04-22T23:00:47.435Z] "- - -" 0 UH - - "-" 0 0 0 - "-" "-" "-" "-" "-" BlackHoleCluster - 172.21.0.43:18500 172.21.0.39:48084 - -
[pod/neuvector-cert-upgrader-job-s9mvl/istio-proxy] [2025-04-22T23:00:52.880Z] "- - -" 0 UH - - "-" 0 0 0 - "-" "-" "-" "-" "-" BlackHoleCluster - 172.21.0.43:18500 172.21.0.39:44924 - -
[pod/neuvector-cert-upgrader-job-s9mvl/istio-proxy] [2025-04-22T23:00:57.970Z] "- - -" 0 UH - - "-" 0 0 0 - "-" "-" "-" "-" "-" BlackHoleCluster - 172.21.0.43:18500 172.21.0.39:44928 - -
[pod/neuvector-cert-upgrader-job-s9mvl/istio-proxy] [2025-04-22T23:01:03.108Z] "- - -" 0 UH - - "-" 0 0 0 - "-" "-" "-" "-" "-" BlackHoleCluster - 172.21.0.43:18500 172.21.0.39:37196 - -
[pod/neuvector-cert-upgrader-job-s9mvl/istio-proxy] [2025-04-22T23:01:08.262Z] "- - -" 0 UH - - "-" 0 0 0 - "-" "-" "-" "-" "-" BlackHoleCluster - 172.21.0.43:18500 172.21.0.39:37202 - -
[pod/neuvector-cert-upgrader-job-s9mvl/istio-proxy] [2025-04-22T23:01:13.507Z] "- - -" 0 UH - - "-" 0 0 0 - "-" "-" "-" "-" "-" BlackHoleCluster - 172.21.0.43:18500 172.21.0.39:53508 - -
[pod/neuvector-cert-upgrader-job-s9mvl/istio-proxy] [2025-04-22T23:01:18.830Z] "- - -" 0 UH - - "-" 0 0 0 - "-" "-" "-" "-" "-" BlackHoleCluster - 172.21.0.43:18500 172.21.0.39:53514 - -
[pod/neuvector-cert-upgrader-job-s9mvl/istio-proxy] [2025-04-22T23:01:18.929Z] "- - -" 0 UH - - "-" 0 0 0 - "-" "-" "-" "-" "-" BlackHoleCluster - 172.21.0.43:18500 172.21.0.39:53528 - -
[pod/neuvector-cert-upgrader-job-s9mvl/istio-proxy] [2025-04-22T23:01:24.151Z] "- - -" 0 UH - - "-" 0 0 0 - "-" "-" "-" "-" "-" BlackHoleCluster - 172.21.0.43:18500 172.21.0.39:49836 - -
[pod/neuvector-cert-upgrader-job-s9mvl/istio-proxy] [2025-04-22T23:01:29.639Z] "- - -" 0 UH - - "-" 0 0 0 - "-" "-" "-" "-" "-" BlackHoleCluster - 172.21.0.43:18500 172.21.0.39:49848 - -
[pod/neuvector-cert-upgrader-job-s9mvl/istio-proxy] [2025-04-22T23:01:34.785Z] "- - -" 0 UH - - "-" 0 0 0 - "-" "-" "-" "-" "-" BlackHoleCluster - 172.21.0.43:18500 172.21.0.39:58660 - -
[pod/neuvector-cert-upgrader-job-s9mvl/istio-proxy] [2025-04-22T23:01:40.155Z] "- - -" 0 UH - - "-" 0 0 0 - "-" "-" "-" "-" "-" BlackHoleCluster - 172.21.0.43:18500 172.21.0.39:58670 - -
[pod/neuvector-cert-upgrader-job-s9mvl/istio-proxy] [2025-04-22T23:01:45.265Z] "- - -" 0 UH - - "-" 0 0 0 - "-" "-" "-" "-" "-" BlackHoleCluster - 172.21.0.43:18500 172.21.0.39:44388 - -
[pod/neuvector-cert-upgrader-job-s9mvl/istio-proxy] [2025-04-22T23:01:50.679Z] "- - -" 0 UH - - "-" 0 0 0 - "-" "-" "-" "-" "-" BlackHoleCluster - 172.21.0.43:18500 172.21.0.39:44394 - -
[pod/neuvector-cert-upgrader-job-s9mvl/istio-proxy] [2025-04-22T23:01:56.069Z] "- - -" 0 UH - - "-" 0 0 0 - "-" "-" "-" "-" "-" BlackHoleCluster - 172.21.0.43:18500 172.21.0.39:56410 - -
[pod/neuvector-cert-upgrader-job-s9mvl/istio-proxy] [2025-04-22T23:02:01.477Z] "- - -" 0 UH - - "-" 0 0 0 - "-" "-" "-" "-" "-" BlackHoleCluster - 172.21.0.43:18500 172.21.0.39:56424 - -
[pod/neuvector-cert-upgrader-job-s9mvl/istio-proxy] [2025-04-22T23:02:06.537Z] "- - -" 0 UH - - "-" 0 0 0 - "-" "-" "-" "-" "-" BlackHoleCluster - 172.21.0.43:18500 172.21.0.39:36686 - -
[pod/neuvector-cert-upgrader-job-s9mvl/istio-proxy] [2025-04-22T23:02:11.793Z] "- - -" 0 UH - - "-" 0 0 0 - "-" "-" "-" "-" "-" BlackHoleCluster - 172.21.0.43:18500 172.21.0.39:36698 - -
[pod/neuvector-cert-upgrader-job-s9mvl/istio-proxy] [2025-04-22T23:02:17.045Z] "- - -" 0 UH - - "-" 0 0 0 - "-" "-" "-" "-" "-" BlackHoleCluster - 172.21.0.43:18500 172.21.0.39:56738 - -
[pod/neuvector-cert-upgrader-job-s9mvl/istio-proxy] [2025-04-22T23:02:22.219Z] "- - -" 0 UH - - "-" 0 0 0 - "-" "-" "-" "-" "-" BlackHoleCluster - 172.21.0.43:18500 172.21.0.39:56740 - -
[pod/neuvector-cert-upgrader-job-s9mvl/istio-proxy] [2025-04-22T23:02:27.550Z] "- - -" 0 UH - - "-" 0 0 0 - "-" "-" "-" "-" "-" BlackHoleCluster - 172.21.0.43:18500 172.21.0.39:42354 - -
[pod/neuvector-cert-upgrader-job-s9mvl/istio-proxy] [2025-04-22T23:02:32.732Z] "- - -" 0 UH - - "-" 0 0 0 - "-" "-" "-" "-" "-" BlackHoleCluster - 172.21.0.43:18500 172.21.0.39:43176 - -
[pod/neuvector-cert-upgrader-job-s9mvl/istio-proxy] [2025-04-22T23:02:38.110Z] "- - -" 0 UH - - "-" 0 0 0 - "-" "-" "-" "-" "-" BlackHoleCluster - 172.21.0.43:18500 172.21.0.39:43192 - -
[pod/neuvector-cert-upgrader-job-s9mvl/istio-proxy] [2025-04-22T23:02:43.311Z] "- - -" 0 UH - - "-" 0 0 0 - "-" "-" "-" "-" "-" BlackHoleCluster - 172.21.0.43:18500 172.21.0.39:39190 - -
[pod/neuvector-cert-upgrader-job-s9mvl/istio-proxy] [2025-04-22T23:02:48.690Z] "- - -" 0 UH - - "-" 0 0 0 - "-" "-" "-" "-" "-" BlackHoleCluster - 172.21.0.43:18500 172.21.0.39:39198 - -
[pod/neuvector-cert-upgrader-job-s9mvl/istio-proxy] [2025-04-22T23:02:53.929Z] "- - -" 0 UH - - "-" 0 0 0 - "-" "-" "-" "-" "-" BlackHoleCluster - 172.21.0.43:18500 172.21.0.39:57946 - -
[pod/neuvector-cert-upgrader-job-s9mvl/istio-proxy] [2025-04-22T23:02:59.136Z] "- - -" 0 UH - - "-" 0 0 0 - "-" "-" "-" "-" "-" BlackHoleCluster - 172.21.0.43:18500 172.21.0.39:57960 - -
[pod/neuvector-cert-upgrader-job-s9mvl/istio-proxy] [2025-04-22T23:02:59.161Z] "- - -" 0 UH - - "-" 0 0 0 - "-" "-" "-" "-" "-" BlackHoleCluster - 172.21.0.43:18500 172.21.0.39:57970 - -
[pod/neuvector-cert-upgrader-job-s9mvl/istio-proxy] [2025-04-22T23:03:04.644Z] "- - -" 0 UH - - "-" 0 0 0 - "-" "-" "-" "-" "-" BlackHoleCluster - 172.21.0.43:18500 172.21.0.39:53286 - -
[pod/neuvector-cert-upgrader-job-s9mvl/istio-proxy] [2025-04-22T23:03:09.968Z] "- - -" 0 UH - - "-" 0 0 0 - "-" "-" "-" "-" "-" BlackHoleCluster - 172.21.0.43:18500 172.21.0.39:53292 - -
[pod/neuvector-cert-upgrader-job-s9mvl/istio-proxy] [2025-04-22T23:03:15.036Z] "- - -" 0 UH - - "-" 0 0 0 - "-" "-" "-" "-" "-" BlackHoleCluster - 172.21.0.43:18500 172.21.0.39:60560 - -
[pod/neuvector-cert-upgrader-job-s9mvl/istio-proxy] [2025-04-22T23:03:20.289Z] "- - -" 0 UH - - "-" 0 0 0 - "-" "-" "-" "-" "-" BlackHoleCluster - 172.21.0.43:18500 172.21.0.39:60572 - -
[pod/neuvector-cert-upgrader-job-s9mvl/istio-proxy] [2025-04-22T23:03:25.659Z] "- - -" 0 UH - - "-" 0 0 0 - "-" "-" "-" "-" "-" BlackHoleCluster - 172.21.0.43:18500 172.21.0.39:44438 - -
[pod/neuvector-cert-upgrader-job-s9mvl/istio-proxy] [2025-04-22T23:03:30.855Z] "- - -" 0 UH - - "-" 0 0 0 - "-" "-" "-" "-" "-" BlackHoleCluster - 172.21.0.43:18500 172.21.0.39:44440 - -
[pod/neuvector-cert-upgrader-job-s9mvl/istio-proxy] [2025-04-22T23:03:35.893Z] "- - -" 0 UH - - "-" 0 0 0 - "-" "-" "-" "-" "-" BlackHoleCluster - 172.21.0.43:18500 172.21.0.39:45252 - -
[pod/neuvector-cert-upgrader-job-s9mvl/istio-proxy] [2025-04-22T23:03:41.059Z] "- - -" 0 UH - - "-" 0 0 0 - "-" "-" "-" "-" "-" BlackHoleCluster - 172.21.0.43:18500 172.21.0.39:45258 - -
[pod/neuvector-cert-upgrader-job-s9mvl/istio-proxy] [2025-04-22T23:03:46.155Z] "- - -" 0 UH - - "-" 0 0 0 - "-" "-" "-" "-" "-" BlackHoleCluster - 172.21.0.43:18500 172.21.0.39:45634 - -
[pod/neuvector-cert-upgrader-job-s9mvl/istio-proxy] [2025-04-22T23:03:51.245Z] "- - -" 0 UH - - "-" 0 0 0 - "-" "-" "-" "-" "-" BlackHoleCluster - 172.21.0.43:18500 172.21.0.39:45644 - -
[pod/neuvector-cert-upgrader-job-s9mvl/istio-proxy] [2025-04-22T23:03:56.507Z] "- - -" 0 UH - - "-" 0 0 0 - "-" "-" "-" "-" "-" BlackHoleCluster - 172.21.0.43:18500 172.21.0.39:59406 - -
[pod/neuvector-cert-upgrader-job-s9mvl/istio-proxy] [2025-04-22T23:04:01.717Z] "- - -" 0 UH - - "-" 0 0 0 - "-" "-" "-" "-" "-" BlackHoleCluster - 172.21.0.43:18500 172.21.0.39:59412 - -
[pod/neuvector-cert-upgrader-job-s9mvl/istio-proxy] [2025-04-22T23:04:06.729Z] "- - -" 0 UH - - "-" 0 0 0 - "-" "-" "-" "-" "-" BlackHoleCluster - 172.21.0.43:18500 172.21.0.39:59358 - -
[pod/neuvector-cert-upgrader-job-s9mvl/istio-proxy] [2025-04-22T23:04:12.167Z] "- - -" 0 UH - - "-" 0 0 0 - "-" "-" "-" "-" "-" BlackHoleCluster - 172.21.0.43:18500 172.21.0.39:59360 - -

Requests

Guidance on need for an internal cert when running within a mesh.

Adding the healthcheck port to controller-service.yaml

[daxmc99]

See our issue here: https://repo1.dso.mil/big-bang/product/packages/neuvector/-/issues/179

I was able to reproduce the issue.

kubectl get pods -n neuvector
NAME                                        READY   STATUS    RESTARTS        AGE
neuvector-cert-upgrader-job-bphmc           1/2     Error     0               3m57s
neuvector-controller-pod-8588d96d6d-7dlcs   1/2     Running   1 (3m52s ago)   4m11s
neuvector-enforcer-pod-jqps7                2/2     Running   0               4m11s
neuvector-enforcer-pod-lpwhj                2/2     Running   0               4m11s
neuvector-manager-pod-f45664f9-579w8        2/2     Running   0               4m11s
neuvector-scanner-pod-5c8c9d49b4-4272d      2/2     Running   1 (3m58s ago)   4m11s

kubectl logs -n neuvector neuvector-cert-upgrader-job-bphmc 
time="2025-08-02T09:30:25Z" level=info msg="Getting running namespace"
time="2025-08-02T09:30:25Z" level=info msg="Creating k8s client"
time="2025-08-02T09:30:25Z" level=info msg="Checking k8s permissions"
time="2025-08-02T09:30:25Z" level=fatal msg="Failed to run the command" error="Post \"https://10.245.0.1:443/apis/authorization.k8s.io/v1/selfsubjectaccessreviews\": dial tcp 10.245.0.1:443: connect: connection refused"`

Controller pod logs:

kubectl logs -n neuvector neuvector-controller-pod-8588d96d6d-7dlcs 
2025-08-02T09:41:08|MON|/usr/local/bin/monitor starts, pid=1
2025-08-02T09:41:08|MON|Start ctrl, pid=7
2025-08-02T09:41:08|MON|Start opa, pid=8
{"level":"warning","msg":"OPA running with uid or gid 0. Running OPA with root privileges is not recommended.","time":"2025-08-02T09:41:09Z"}
2025-08-02T09:41:09.285|INFO|CTL|main.main: START - version=v5.4.5
2025-08-02T09:41:09.285|INFO|CTL|main.main: - join=neuvector-svc-controller.neuvector
2025-08-02T09:41:09.286|INFO|CTL|main.main: - advertise=10.244.0.239
2025-08-02T09:41:09.286|INFO|CTL|main.main: - bind=10.244.0.239
2025-08-02T09:41:09.292|INFO|CTL|container.Connect: - endpoint=
2025-08-02T09:41:09.293|INFO|CTL|container.InitStubRtDriver:
2025-08-02T09:41:09.378|ERRO|CTL|orchestration.GetK8sVersion: getVersion fail - code=403, tag=oc, useToken=true
2025-08-02T09:41:09.378|INFO|CTL|main.main: - cgroups=2
2025-08-02T09:41:09.378|INFO|CTL|main.main: Container socket connected - endpoint= runtime=stubRtDriver
2025-08-02T09:41:09.401|INFO|CTL|resource.IsRancherFlavor: resource found - namespace=cattle-system service=cattle-cluster-agent
2025-08-02T09:41:09.401|INFO|CTL|main.main: - flavor=Rancher k8s=1.32.5 oc=
2025-08-02T09:41:09.583|INFO|CTL|main.main: - host={ID:devendra-tf-k8s-worker-pool-lhf1b:6f4faa8b-8d33-422e-b858-caacd49d16f7 Name:devendra-tf-k8s-worker-pool-lhf1b Runtime:stubRtDriver Platform:Kubernetes Flavor:Rancher CloudPlatform: Network:Default RuntimeVer:1.0 RuntimeAPIVer:1.0 OS:SUSE Linux Enterprise Server 15 SP7 Kernel:6.1.0-35-amd64 CPUs:8 Memory:16767778816 Ifaces:map[] TunnelIP:[] CapDockerBench:false CapKubeBench:false StorageDriver: CgroupVersion:2}
2025-08-02T09:41:09.583|INFO|CTL|main.main: - ctrler={CLUSDevice:{ID:c7bf232c95ff03a5beb18aace7385d075e339778e9cc4266614281900557dfa8 Name:k8s_neuvector-controller-pod_neuvector-controller-pod-8588d96d6d-7dlcs_neuvector_6a4434b3-3db6-4e23-9d95-142fbe8bc554_0 SelfHostname: HostName:devendra-tf-k8s-worker-pool-lhf1b HostID:devendra-tf-k8s-worker-pool-lhf1b:6f4faa8b-8d33-422e-b858-caacd49d16f7 Domain:neuvector NetworkMode:default PidMode: Ver:v5.4.5 Labels:map[app:neuvector-controller-pod io.kubernetes.container.name:neuvector-controller-pod io.kubernetes.pod.name:neuvector-controller-pod-8588d96d6d-7dlcs io.kubernetes.pod.namespace:neuvector io.kubernetes.pod.uid:6a4434b3-3db6-4e23-9d95-142fbe8bc554 name:controller neuvector.role:controller pod-template-hash:8588d96d6d release: security.istio.io/tlsMode:istio service.istio.io/canonical-name:neuvector-controller-pod service.istio.io/canonical-revision:latest topology.istio.io/network:network1 vendor:NeuVector Inc. version:] CreatedAt:2025-08-02 09:41:09.293521832 +0000 UTC StartedAt:2025-08-02 09:41:09.293521832 +0000 UTC JoinedAt:0001-01-01 00:00:00 +0000 UTC MemoryLimit:0 CPUs: ClusterIP: RPCServerPort:0 Pid:1 Ifaces:map[eth0:[{IPNet:{IP:10.244.0.239 Mask:ffffffff} Gateway: Scope:global NetworkID: NetworkName:}]]} Leader:false OrchConnStatus: OrchConnLastError: ReadPrimeConfig:false}
2025-08-02T09:41:09.584|INFO|CTL|main.main: start initializing k8s internal secret controller and wait for internal secret creation if it's not created
2025-08-02T09:41:09.681|INFO|CTL|migration.(*InternalSecretController).ReloadSecret: internal certificate is not ready yet.

To me it looks like cert-upgrader pod is not able to communicate to kubernetes api server, so the healthcheck port 18500 will come later into picture. Here the communication between cert-upgrader-pod is blocked to kube-apiserver

As a workaround, you can edit the cronjob/neuvector-cert-upgrader-pod to add a annotation sidecar.istio.io/inject: "false" in spec.jobTemplate.spec.template.annotations

spec:
  concurrencyPolicy: Forbid
  failedJobsHistoryLimit: 3
  jobTemplate:
    metadata:
      creationTimestamp: null
    spec:
      activeDeadlineSeconds: 3600
      backoffLimit: 6
      completions: 1
      parallelism: 1
      template:
        metadata:
          annotations:
            sidecar.istio.io/inject: "false"       # <<<<THIS LINE
          creationTimestamp: null
          labels:
            app: neuvector-cert-upgrader-pod
            release: neuvector

The helm template for upgrader-cronjob.yaml already provides a way to add annotations in values.yaml

spec:
      activeDeadlineSeconds: {{ .Values.controller.certupgrader.timeout }}
      parallelism: 1
      completions: 1
      backoffLimit: 6
      template:
        metadata:
          labels:
            app: neuvector-cert-upgrader-pod
            release: {{ .Release.Name }}
            {{- with .Values.controller.certupgrader.podLabels }}
            {{- toYaml . | nindent 12 }}
            {{- end }}
          {{- with .Values.controller.certupgrader.podAnnotations }}
          annotations:
          {{- toYaml . | nindent 12 }}
          {{- end }}

So your values.yaml should contain below values:

controller:
  certupgrader:
    podAnnotations:
      sidecar.istio.io/inject: "false"

[daxmc99]

True this is valid workaround but it negates the security requirements when running in a cluster where Istio is required for all connections.

I will try to make an MR to solve this at the chart level

@daxmc99 Are you able to reproduce this issue with the latest version of NeuVector?
I am unable to get the blackhole logs on the sidecar container. The only logs I see is that the communication to kube-apiserver is broken, and disabling sidecar injection to neuvector namespace fixes this as a workaround.