Options for real HA egress

[tschuering]

As far as I understand the current egress feature is a static defined floating IP or static node IP in one Cilium NAT policy, which is linked to the egress node defined in the agent_nodepools. You have well the option to create multiple egress nodes and therefore multiple egress policies and spread your application for the example in your deployment on the different egress policies (COMPLEX!), but I do not see any option to have real HA in that case, that I have the situation where I have one pod or namespace and I want to make sure that always egress is available. It is always pointed to the one static egress IP. Correct me if I am wrong? And sadly Cilium does not support HA egress itself in OSS, only in Enterprise version, where you can define multiple IPs in some sort of pools.

I had another use-case in a non-Kubernetes environment, where I actually managed to implement something like this via KeepAlived and floating IP. So if there is not yet a better option available, which I did not yet find here, there is a way I thought.

So I simulated some ideas:

First:

The first idea I had, was using one floating IP between three egress nodes and detach and attach that manually to simulate some KeepAlived behaviour, to see that happens. Apparently Cilium does not like this at all. It works initially when I attached one floating IP manually on one to the egress nodes and attach the CNI policy to it with the floating IP. But after I switched the floating IP to another node, it did not work anymore. Restart of Cilium did not help. There is some sort of caching in between or something I miss totally in Cilium, i do not know yet, but I am also not the expert at all in Cilium. But it just did not work. After I switched back the floating IP to the first node I attached it, it worked again. And yes I had manually added the IPs and deleted them in the process on the egress nodes itself by "ip add/ip del".

Second:

So I thought, okay why using single floating IP at all, cause a switchover later via KeepAlived would cost unnecessary time. I am using KeepAlived and floating IPs in non-k8s setups, and it costs always time. I even needed to write a daemon to make sure that the IPs are always correctly assigned to the KeepAlived MASTER node, because Hetzner tend to like dumb things when switching floating IPs sometimes via a stupid curl script to the API. So why just implement some KeepAlived behaviour not on the IP layer but on the Policy layer. So I just used three egress nodes in that case without floating IP (would be the same with), and just change manually the egressIP defined in the policy.yml. Also did not work, only one of the three actually worked and I had no clue why. Maybe some hint here?

Third:

I just thought maybe I need to switch more than just the IP, so that Cilium would actually see that somethings changes. So I deployed two egress nodes, but all sitting in their own pool and with label annotation egress-1, egress-2 instead both using egress only. And now I did a switchero via 2 different policy.yml files and now magically it starts to switch fine without any issues. See screenshot below. I mean, with that knowledge first idea basically should also work, but why make it complex and SPOF with one floating IP, which might also not work at Hetzners side at one point, so stick to the 2 or 3 egress node with own IP setup.

Simulated solution?

Whats next?

There are some options here. The first one is the dirty "simple" one, and the second could be the more mature one.

First

Use Kubernetes cronjob and some RBAC permissions for run the cronjob linked to the service account to actually check every X minutes whats the status of the egress nodes is, and do switcherooo if needed.

Second

In Packer?:

• Install KeepAlived and make sure that it is disabled by default via "systemctl disable"

In Terraform:

1. Create an option to actually define the feature as a config parameter for kube.tf. With that enabled the KeepAlived service gets started on all egress nodes and with unicast communication (requirement in Hetzner Cloud internal network) all egress gateways which share some common variable spread over three egress pools will communicate via KeepAlived don't use any virtual IP or sth at all (You can do that), but just the Master defines that he will be doing kubectl apply his egress.yml, which could be stored in a ConfigMap or whatever. The others are standby.

@mysticaltech I am open for some other ideas or maybe I totally oversaw something essential, but I want basically to not go the Proxy (See in README -> "Deploy in a pre-constructed private network (for proxies etc") or NAT gateway way, as it would create the need in our company use case to deploy those via Ansible, as we do that for all non-k8s deployments currently. And also it is some non-k8s, k8s mixture I personally do not like. Proxy is also not an option cause it is in that case it is HTTP/HTTPS only.

[mysticaltech]

@tschuering Thanks for those ideas. I actually did not implement the cilium egress feature myself. So I would need to dive deeper to be able to help you. I will have a look ASAP.

In the meantime, @M4t7e maybe you could help here as you know cilium much better.

[maaft]

I'd also be interested in an integrated HA solution to this!

[M4t7e]

@tschuering Have you considered not using a dedicated egress gateway? That’s the best HA setup, with each node having its own internet connection, and I prefer it whenever possible 🙂

If you still need a dedicated egress gateway, your third option seems the most promising. It should work, and you can also automate it with keepalived (and custom scripts) or other tools.

Unfortunately, I currently have no better solution for implementing this with the free version of Cilium.

[tschuering]

@M4t7e I also like that more, but we have some situations where we do not have plannable nodes (eg. Autoscaling) but need dedicated access through an egress to other non-k8s services which have IPs whitelisted. So you cannot always work without egress, especially if you are using a dynamic autocluster with Autoscaling.

I will test the third way of implement this in the following weeks at some points, but do not expect anything the next 3 months, as I have too much other work on the table at the moment, and the current setup is suitable enough and the SLA accepts some downtime, if egress would be down for some minutes because of Autoupdate for example.

I will report back here then.