k3s with ufw on ubuntu

[zerr0s]

Hello,

I'm trying to use k3s in a lab before deploying in a producton environment.

All my nodes are dedicated servers in cloud and they have 2 nics. One public and one local.
I have to configure ufw to block all requests in the public interface and allow some ports.

I have read a lot of things over the net and I have setup these rules on ufw on all nodes

Status: active

     To                         Action      From
     --                         ------      ----
[ 1] 22 on ens32                ALLOW IN    Anywhere                  
[ 2] Anywhere on ens33          ALLOW IN    172.20.20.1               
[ 3] Anywhere on ens33          ALLOW IN    172.20.20.2               
[ 4] Anywhere on ens33          ALLOW IN    172.20.20.3               
[ 5] 6443 on ens32              ALLOW IN    Anywhere                  
[ 6] Anywhere                   ALLOW IN    10.42.0.0/16              
[ 7] Anywhere                   ALLOW IN    10.43.0.0/16              
[ 8] 80 on ens32                ALLOW IN    Anywhere                  
[ 9] 443 on ens32               ALLOW IN    Anywhere                  
[10] 22 (v6) on ens32           ALLOW IN    Anywhere (v6)             
[11] 6443 (v6) on ens32         ALLOW IN    Anywhere (v6)             
[12] 8 (v6) on ens32            ALLOW IN    Anywhere (v6)             
[13] 80 (v6) on ens32           ALLOW IN    Anywhere (v6)             
[14] 443 (v6) on ens32          ALLOW IN    Anywhere (v6)                        

172.20.20.1/2/3 are my nodes local IP adress (ens33). 10.42 and 10.43 are the cidr and svv-cidr used during the first install.
ens32 is the public network interface used to permit public requests.

NAME   STATUS   ROLES                       AGE   VERSION        INTERNAL-IP   EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION      CONTAINER-RUNTIME
k3s1   Ready    control-plane,etcd,master   27m   v1.24.6+k3s1   172.20.20.1   x.x.x.x   Ubuntu 22.04.1 LTS   5.15.0-48-generic   containerd://1.6.8-k3s1
k3s2   Ready    control-plane,etcd,master   25m   v1.24.6+k3s1   172.20.20.2   y.y.y.y   Ubuntu 22.04.1 LTS   5.15.0-48-generic   containerd://1.6.8-k3s1
k3s3   Ready    control-plane,etcd,master   24m   v1.24.6+k3s1   172.20.20.3   z.z.z.z   Ubuntu 22.04.1 LTS   5.15.0-48-generic   containerd://1.6.8-k3s1

The traefik svc are listening on all nodes and redirect to the traefik pod:

NAMESPACE     NAME             TYPE           CLUSTER-IP     EXTERNAL-IP                           PORT(S)                      AGE   SELECTOR
default       kubernetes       ClusterIP      10.43.0.1      <none>                                443/TCP                      28m   <none>
kube-system   kube-dns         ClusterIP      10.43.0.10     <none>                                53/UDP,53/TCP,9153/TCP       28m   k8s-app=kube-dns
kube-system   metrics-server   ClusterIP      10.43.98.214   <none>                                443/TCP                      28m   k8s-app=metrics-server
kube-system   traefik          LoadBalancer   10.43.211.33   x.x.x.x,y.y.y.y,z.z.z.z   80:31659/TCP,443:32140/TCP   27m   app.kubernetes.io/instance=traefik,app.kubernetes.io/name=traefik

But when I try to access to ports 80 or 443, it's not working:

root@k3s3:/opt/rancher# nc -vw4 x.x.x.x 443
nc: connect to 192.168.2.1 port 443 (tcp) timed out: Operation now in progress
root@k3s3:/opt/rancher# nc -vw4 x.x.x.x 80
nc: connect to 192.168.2.1 port 80 (tcp) timed out: Operation now in progress

[EDIT]: When I have just one node, it is working well. When I add another node and the traefik service is loadbalanced between all nodes, the timeout apperas randomly. I have troed to disable the default traefik and install traefik from helm directly in daemonset mode to force all node to have a traefik instance on ports 80 and 443. But it is not working.

Any idea to solve this please ?

Thanks.

[stale[bot]]

This repository uses a bot to automatically label issues which have not had any activity (commit/comment/label) for 180 days. This helps us manage the community issues better. If the issue is still relevant, please add a comment to the issue so the bot can remove the label and we know it is still valid. If it is no longer relevant (or possibly fixed in the latest release), the bot will automatically close the issue in 14 days. Thank you for your contributions.

[caroline-suse-rancher]

I'm going to convert this to a discussion since this is more of a question than a bug/feature request.

[colinmollenhour]

@zerr0s I'm interested to know what your solution was if you found one? Thanks!