Automatically apply taint during node self-registering if IP is blacklisted

[schlichtanders]

Hi there,
I thought a lot about how to manage the blacklisted IP problem for an autoscaling group. This is my best idea so far:

• there is a kubelet-arg --register-with-taints which can set taints, including the Unschedulable taint, which would prevent pods being scheduled on this node. EDIT: k3s agent has the node-taint config which probably should be preferred.
• somehow it should be possible to check before the start of the k3s server whether the current IP is actually blacklisted. Depending on this the respective kubelet-arg should be set or not
• if the node is immediately registered as unschedulable, the cluster autoscaler would (hopefully more or less immediately) spawn a new node and somewhen evict the broken node.

Open Questions:

• I have no clue how to run custom code before the k3s server starts
• I don't know whether kubelet-arg or anything can refer to Environment Variables or or have other means to set the taints dynamically

[schlichtanders]

Maybe one solution could be to support a k3s_exec_server_args equivalent for the k3s agent, probably to be called k3s_exec_agent_args.

If these exec args would be directly run, they could spawn a shell subprocess to decide whether taints are added or not.

(Note that the current k3s_exec_server_args do not allow for such dynamic code execution, as the args are wrapped into single quotes instead of double quotes.)

[mysticaltech]

@schlichtanders Funny enough there's an active PR to add k3s_exec_agent_args.

[schlichtanders]

Amazing: it is already merged #862

BUT: it uses single quotes... so my goal does not work here... I commented on the pullrequest

[mysticaltech]

@schlichtanders Actually on this cluster at least, I do not see how it would replace the node. The best solution for this problem is just to use a container proxy, see k3s_registries.

[mysticaltech]

This project also has a decent amount of examples of people that solve the problem with this method. It's also what the kubernetes team themselves recommended.

[schlichtanders]

I put this currently on hold, but will need to come back to this.

Yes, I understood that the container proxy works to enable downloading of the containers, however the service I am offering on Hetzner is a bit like a private cloud itself. I am fearing that my customers would run into similar problems with these blacklisted IPs even if I am able to load the container.

There is also the egress solution which would pipe all requests through one fixed node. This would enable the users to circumvent the blacklisted IPs, but will be a performance bottleneck. My cloud is mainly for real-time data analysis, possibly connecting to Kafka. I fear that the standardized egress would cause problems here.

Hence I am really looking for a way to get a new Hetzner node as soon as possible (as this is also what Hetzner recommends), and marking the node as unschedulable could indeed solve this.

[mysticaltech]

@schlichtanders If that's the problem you are encoutering, there is a simple trick, take down the cluster, reserve the same number of static ips as you had nodes, this will lock the ips including the affected one in static ips and will force hetzner to give new ips to all nodes, and hopefully on this second try it will be ok. As it's become rare enough lately. Then you delete the static ips of course. So it doesn't actually cost much.

[schlichtanders]

That is unfortunately not enough. I want the autoscaling group to work seamlessly.

Preallocating IPs is just not feasible.
My mentioned approach is really the best I came up with so far