Add nftables configuration for K3s on Arch Linux

[cioionut]

When using Arch Linux with kernel 6.18+, k3s-ansible enables nftables but uses the default restrictive ruleset, which blocks K3s traffic.

This commit adds:

• A template for nftables.conf with K3s required ports
• A handler to reload nftables when the config changes
• Configuration task to deploy the template after enabling nftables

The template allows:

• SSH (22)
• K3s API server (6443)
• etcd (2379-2380) for HA clusters
• Flannel VXLAN (8472)
• Kubelet (10250)
• Flannel Wireguard (51820, 51821)
• NodePort range (30000-32767)
• Cluster and service CIDRs

Linked Issues

After running the install playbook or upgrade using the latest ArchLinux cloud image, the connection to the k3s api server (port 6443) is blocked by the nftables firewall. So, for the new install/upgrades will be impossible to manage the cluster via kubectl, k9s, etc without a ssh tunnel (ssh 22 is not blocked by the firewall)

[dereknola]

ArchLinux cloud image

By default on most OSes, the nftables start with no configuration and should not be blocking on any ports. Is this a specific issue to the cloud image? Has ArchLinux started shipping with "default restrictive rules"?

[cioionut]

This is what I encountered last week when I tried to upgrade my k3s cluster. Then I tried a fresh setup using the install playbook on the latest archlinux cloud image and have the same restrictions.
You are right, should not be an issue with the default firewall rules because they should be permissive by default. I guess something from prereq enables the firewall, and then some rules block all the traffic except 22.
I'll look more into this over the weekend and get back with more updates. Thank you!

[cioionut]

Hello! I discovered the code responsible for the current issue. Look at lines 35-64, here the nftables is enabled, and this creates a /etc/nftables.conf by default with restrictive rules.

Here is the proof, I ran the commands manually to test this behaviour that's done in the prereq role.

[archuser@w0 ~]$ cat /etc/nftables.conf
cat: /etc/nftables.conf: No such file or directory
[archuser@w0 ~]$ sudo pacman -S iptables-nft nftables
resolving dependencies...
looking for conflicting packages...
:: iptables-nft-1:1.8.11-2 and iptables-1:1.8.11-2 are in conflict. Remove iptables? [y/N] y

Packages (3) iptables-1:1.8.11-2 [removal]  iptables-nft-1:1.8.11-2  nftables-1:1.1.6-2

Total Download Size:   0.83 MiB
Total Installed Size:  3.46 MiB
Net Upgrade Size:      1.10 MiB

:: Proceed with installation? [Y/n] 
:: Retrieving packages...
 nftables-1:1.1.6-2-x86_64                                         433.9 KiB  1955 KiB/s 00:00 [########################################################] 100%
 iptables-nft-1:1.8.11-2-x86_64                                    417.7 KiB  1536 KiB/s 00:00 [########################################################] 100%
 Total (2/2)                                                       851.7 KiB  2.65 MiB/s 00:00 [########################################################] 100%
(2/2) checking keys in keyring                                                                 [########################################################] 100%
(2/2) checking package integrity                                                               [########################################################] 100%
(2/2) loading package files                                                                    [########################################################] 100%
(2/2) checking for file conflicts                                                              [########################################################] 100%
(3/3) checking available disk space                                                            [########################################################] 100%
:: Processing package changes...
(1/1) removing iptables                                                                        [########################################################] 100%
(1/2) installing nftables                                                                      [########################################################] 100%
Optional dependencies for nftables
    python: Python bindings [installed]
    python-jsonschema: Python bindings [installed]
(2/2) installing iptables-nft                                                                  [########################################################] 100%
:: Running post-transaction hooks...
(1/3) Reloading system manager configuration...
(2/3) Restarting marked services...
(3/3) Arming ConditionNeedsUpdate...
[archuser@w0 ~]$ sudo systemctl start nftables
[archuser@w0 ~]$ cat /etc/nftables.conf
#!/usr/bin/nft -f
# vim:set ts=2 sw=2 et:

# IPv4/IPv6 Simple & Safe firewall ruleset.
# More examples in /usr/share/nftables/ and /usr/share/doc/nftables/examples/.

destroy table inet filter
table inet filter {
  chain input {
    type filter hook input priority filter
    policy drop

    ct state invalid drop comment "early drop of invalid connections"
    ct state {established, related} accept comment "allow tracked connections"
    iif lo accept comment "allow from loopback"
    ip protocol icmp accept comment "allow icmp"
    meta l4proto ipv6-icmp accept comment "allow icmp v6"
    tcp dport ssh accept comment "allow sshd"
    pkttype host limit rate 5/second counter reject with icmpx type admin-prohibited
    counter
  }
  chain forward {
    type filter hook forward priority filter
    policy drop
  }
}
[archuser@w0 ~]$ uname -r
6.18.7-arch1-1

[dereknola]

This was added recently with #486.

As a much simpler fix, is it possible to just not start nftables? As long as iptables-nft/nftables package is present, K3s should be able to operate. Since you have a good setup to test this, can you try just removing

    - name: Ensure nftables is enabled and started
      ansible.builtin.systemd:
        name: nftables
        state: started
        enabled: true

Also there is also the --prefer-bundled-bin which allows k3s to use its own version of iptables.
Basically, there shouldn't be a need to configure a bunch of firewall rules for K3s. It doesn't actually use/need the nftables service.

[cioionut]

I tested with the - name: Ensure nftables is enabled and started commented, and with --prefer-bundled-bin and it's working well, but I wonder why for other distros we impose some firewall rules in prereq?
The current roles/prereq/templates/nftables.conf.j2 is trying to replicate the same firewall configs that were initially set up for other distros.

[dereknola]

So two things are happening:

1. What does k3s require? In this instance, k3s need an iptables it can manipulate, due to its use of kube-router as network policy controller. This is why we need the package to be present (or to use prefer-bundled-bin.
2. What do users actual systems look like. Our official recommendation is to not run any firewall and instead rely on k8s networking security for isolation/security. As you can imagine however, alot of people still have/rely on OS firewalls for security. The firewall configs you see for ufw and firewalld are for this case. But crucially this is a "if ufw or firewalld is already active... add rules to let k3s work".

So there are two parts to your PR that I would ask for changes.

1. Don't turn on nftables service. It is not explicitly required for k3s to operate.
2. Setup a new task that "Checks for nftables service" and IF nftables is ALREADY running as a firewall, then deploy the rules you have written.

[gillouche]

Hello,

nftables is only required for podman/docker with kernel >=6.18.

I have a combination of k3s and podman containers in my homelab so I blindly enabled it in the PR, sorry for the regression.

The current nftables.conf of this PR will remove any existing rule that could have been configured outside k3s install.

Instead of overwriting the default nftables.conf (that will be overwritten with any nftables package update), we should have a file for k3s created in the folder loaded by nftables.conf instead: /etc/nftables.d/

We just need to ensure that the following instruction is in nftables.conf:

include "/etc/nftables.d/*.nft"

We shouldn't have a flush ruleset in the custom k3s conf, this should only be done in nftables.conf before any custom rule is loaded.

[cioionut]

Hi @gillouche ! Good catch!
You can check my changes now.

[dereknola]

No need to make this a dedicated handler, its already conditionally controlled via the Block.
Also, don't use systemd we are using the more generic service module for expanded init support unless absolutely required.

[cioionut]

Thank you! I changed that one.

[dereknola]

@cioionut Just need to fix the lint errors:

./roles/prereq/tasks/main.yml
Error: 58:1 [trailing-spaces] trailing spaces