Review of Firewall Rules & Security Consideration

[M4t7e]

Hello guys, I would like to review the current firewall rules and also discuss the security aspects with you. I see a few rules that seem to do nothing and others can also be harmful.

Current Inbound Rules:

No	Name	Network	Protocol	Port
1	Allow Incoming ICMP Ping Requests	Any IPv4 Any IPv6	ICMP	-
2	Allow Incoming Requests to Hubble Server & Hubble Relay (Cilium)	Any IPv4 Any IPv6	TCP	4244-4245
3	Allow Incoming Requests to Kube API Server	Any IPv4 Any IPv6	TCP	6443
4	Allow Internal Cluster TCP Traffic	10.0.0.0/8 127.0.0.1 169.254.169.254 213.239.246.21	TCP	any
5	Allow Internal Cluster UDP Traffic	10.0.0.0/8 127.0.0.1 169.254.169.254 213.239.246.21	UDP	any
6	Allow Incoming SSH Traffic	Any IPv4 Any IPv6	TCP	22

Here my thoughts:

• Rule 1: Call me paranoid, but reality has shown me that even ping can be dangerous (see CVE-2022-23093 -> FreeBSD Remote Code Execution vulnerability in ping). Would it hurt to disable it by default, and if one needs it, it can still be enabled via extra_firewall_rules?
• Rule 2: It seems we expose internal endpoints to the public world (see also Hubble Server & Hubble Relay (Cilium) ports open to the world in FW #777). I would like to remove that rule if nobody sees any good reason to expose it to the world by default.
• Rule 3: This seems to me the biggest threat. The Kube API should better not be exposed to the public internet. It is an unnecessarily exposed attack surface and scanners like Shodan have already all K3s Kube API IPs running on Hetzner in their DB. In the worst case, if a dangerous vulnerability emerges, attackers can easily take control of many clusters, which would be catastrophic.
• Rules 4+5: Not dangerous but seems totally useless:
  • The firewall does not work for the internal network (private & internal IPs are obsolete here -> 10.0.0.0/8 & 127.0.0.1). See https://docs.hetzner.com/cloud/firewalls/faq/#can-firewalls-secure-traffic-to-my-private-hetzner-cloud-networks
  • The meta service (169.254.169.254 -> see https://docs.hetzner.cloud/#server-metadata) can be queried, but will never connect to a node and additionally this is a link-local address and will probably also never be affected by the firewall).
  • The Hetzner API (api.hetzner.cloud -> 213.239.246.21 & 2a01:4f8:0:1::4:21) will never initiate a connection to a node. The only way that makes sense is the other way around.
• Rule 6: Also best practice here to not allow SSH connection from everywhere.

I think rules 1,2,4 and 5 can be deleted with no harm and no loss of functionality. Rules 3 and 6 are required for deployment, but what if we make it possible to optionally configure a static IP/network to access it from? For Rule 3, I would also prefer not to expose it to the public at all (the Kube API is a big attack surface and also just a small misconfiguration can have dramatic consequences), but I don't know if there is a suitable solution to achieve that?

Would anyone like to contribute their opinion on this?

[mysticaltech]

@M4t7e Super insightful suggestions. @kube-hetzner/core FYI.

I think we can do that, PR most welcome, but if possible in a backward compatible fashion for the rules that are actively used now, as for the others like 4 and 5, we can remove them, that makes much sense indeed!

As for # 6, maybe we can whitlist only the deploying IP, but also have a variable that accepts a list of IPs, that way, if our IP changes, we can change that value, apply, and it will update the firewall for us to whitelist our current IP.

Lastly for # 3, it's taken care of by # 6, because the kube-api can be accessed from the control planes directly just by SSH'ing and using kubectl inside any control-plane node.

[M4t7e]

Thanks for your reply!

I tested the removal yesterday with different scenarios and everything seems to be fine. I would skip the removal of rule 1, because there is already a setting (block_icmp_ping_in) to disable it. Haven't seen that before...

For rule 6 I have no good idea how to solve that. Dynamically obtaining the public IP can be hard and depending on the network setup of the client, it can also be unstable. In some network setups this could lead to weird situations if the public IP can vary over time -> CGNAT, public IPv6 (lease times & PEX), mobile Hotspots, etc. I think this is only a good solution if you connect via a static IP (e.g. a bastion/jump-server or a VPN with stable IPs).

I would create a PR for removal of rules 2,3,4 and 5 if you think that is okay. It is backwards compatible in the sense that it won't break the cluster. Everything internally will work fine. Only users who are connecting to Kube API via the public IP would see an issue. For those I can add a section to the readme (maybe close to that section: https://github.com/kube-hetzner/terraform-hcloud-kube-hetzner/tree/master#-installation) with an explanation and an example in the kube.tf how to expose it again, together with a hint that there are safer options to do that.

For production setups, I think users would probably use Flux or Argo CD and only connect to the control plane for administrative tasks. For users that mostly work with kubectl on the CP Nodes, we could also add the kubectl bash completion (I can also add a PR for that) to make the life a bit easier working there.

TLDR:

• I would remove rules 2,3,4 and 5
• I would add a section to the documentation and an example to kube.tf how to reach the Kube API again
• I would add the kubectl bash completion for a nicer interaction on the CP Nodes with a new PR
• The SSH Port will stay public available until there is a better and stable option

What do you think? Could this be a way forward?

[mysticaltech]

Yes @StefanIGit, absolutely, anyone wants to create a PR for this, don't hesitate, it should be simple enough.

[mysticaltech]

Otherwise, I will see if I can find the time for that ASAP.

[M4t7e]

Thanks for the discussion guys! I can create a PR for that, but earliest at the beekend, because I am currently on a business trip. I would delete the obsolete rules and try to implement the automatic IP update for the ssh rule.

[mysticaltech]

Sounds good!! 👍

[M4t7e]

I added 2 PRs, one for the firewall cleanup (#896) and the other to add kubectl bash autocompletion (#895)

[M4t7e]

Thx for your support guys! 🙂