k3s with ufw on ubuntu #6240
Description
Hello,
I'm trying to use k3s in a lab before deploying in a producton environment.
All my nodes are dedicated servers in cloud and they have 2 nics. One public and one local.
I have to configure ufw to block all requests in the public interface and allow some ports.
I have read a lot of things over the net and I have setup these rules on ufw on all nodes
Status: active
To Action From
-- ------ ----
[ 1] 22 on ens32 ALLOW IN Anywhere
[ 2] Anywhere on ens33 ALLOW IN 172.20.20.1
[ 3] Anywhere on ens33 ALLOW IN 172.20.20.2
[ 4] Anywhere on ens33 ALLOW IN 172.20.20.3
[ 5] 6443 on ens32 ALLOW IN Anywhere
[ 6] Anywhere ALLOW IN 10.42.0.0/16
[ 7] Anywhere ALLOW IN 10.43.0.0/16
[ 8] 80 on ens32 ALLOW IN Anywhere
[ 9] 443 on ens32 ALLOW IN Anywhere
[10] 22 (v6) on ens32 ALLOW IN Anywhere (v6)
[11] 6443 (v6) on ens32 ALLOW IN Anywhere (v6)
[12] 8 (v6) on ens32 ALLOW IN Anywhere (v6)
[13] 80 (v6) on ens32 ALLOW IN Anywhere (v6)
[14] 443 (v6) on ens32 ALLOW IN Anywhere (v6)
172.20.20.1/2/3 are my nodes local IP adress (ens33). 10.42 and 10.43 are the cidr and svv-cidr used during the first install.
ens32 is the public network interface used to permit public requests.
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
k3s1 Ready control-plane,etcd,master 27m v1.24.6+k3s1 172.20.20.1 x.x.x.x Ubuntu 22.04.1 LTS 5.15.0-48-generic containerd://1.6.8-k3s1
k3s2 Ready control-plane,etcd,master 25m v1.24.6+k3s1 172.20.20.2 y.y.y.y Ubuntu 22.04.1 LTS 5.15.0-48-generic containerd://1.6.8-k3s1
k3s3 Ready control-plane,etcd,master 24m v1.24.6+k3s1 172.20.20.3 z.z.z.z Ubuntu 22.04.1 LTS 5.15.0-48-generic containerd://1.6.8-k3s1
The traefik svc are listening on all nodes and redirect to the traefik pod:
NAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR
default kubernetes ClusterIP 10.43.0.1 <none> 443/TCP 28m <none>
kube-system kube-dns ClusterIP 10.43.0.10 <none> 53/UDP,53/TCP,9153/TCP 28m k8s-app=kube-dns
kube-system metrics-server ClusterIP 10.43.98.214 <none> 443/TCP 28m k8s-app=metrics-server
kube-system traefik LoadBalancer 10.43.211.33 x.x.x.x,y.y.y.y,z.z.z.z 80:31659/TCP,443:32140/TCP 27m app.kubernetes.io/instance=traefik,app.kubernetes.io/name=traefik
But when I try to access to ports 80 or 443, it's not working:
root@k3s3:/opt/rancher# nc -vw4 x.x.x.x 443
nc: connect to 192.168.2.1 port 443 (tcp) timed out: Operation now in progress
root@k3s3:/opt/rancher# nc -vw4 x.x.x.x 80
nc: connect to 192.168.2.1 port 80 (tcp) timed out: Operation now in progress
[EDIT]: When I have just one node, it is working well. When I add another node and the traefik service is loadbalanced between all nodes, the timeout apperas randomly. I have troed to disable the default traefik and install traefik from helm directly in daemonset mode to force all node to have a traefik instance on ports 80 and 443. But it is not working.
Any idea to solve this please ?
Thanks.