Ingress traffic Contour on our Talos clusters is not working anymore when Omni server is down.

[peterbosalliandercom]

Hi all,
We’re running into a strange issue and could use some insights. During a release this weekend, our Omni server was brought down. About 30 seconds after that, all ingress traffic to our Talos clusters suddenly stopped.
For context: we're using Contour/Envoy as our ingress controller.
We’re a bit in the dark about what could cause this behavior. Has anyone seen something similar, or does anyone have ideas about what might trigger a complete halt of ingress traffic after an upstream system goes down?
Thanks in advance for any thoughts or directions to investigate!

[Unix4ever]

Hi,
Omni being down usually doesn't affect ingress.

Running Omni is critical only in two cases:

• you run your cluster without KubePrism enabled. Then Omni becomes the loadbalancer for the worker nodes when they go to Kubernetes API on the control planes. KubePrism fixes that as it adds direct node-to-node link from workes to controlplanes.
• when you try to use kubectl, talosctl: it's proxied through Omni, so you lose the access.

So my only guess is that ingress pods could start crashlooping because you have KubePrism disabled.
KubePrism enabled for all clusters above 1.6. If Talos cluster was created before 1.6, you might need to enable it explicitly through the machine config patch. Config example.

[peterbosalliandercom]

We checked, kubeprism is enabled. We are above 1.6. (these are newly created clusters)

[peterbosalliandercom]

[smira]

Make sure your Countour/Envoy uses KubePrism endpoint for the controlplane endpoint (or Kubernetes service).