K3S metrics server connection timed out

[fedesysadmin]

Hello folks,
I deployed K3S in my virtualization environement (vmware based).
I installed K3S cluster using the airgap mode and I am using ETCD database.

This is the current scenario:

• 1 HAProxy for balacing external requests (nginx pod is reachble via NodePort configuration)
• 3 K3S hosts. These hosts are master+worker.

kubectl get nodes
NAME  STATUS   ROLES                  AGE   VERSION
k3s-1   Ready    control-plane,etcd   9d    v1.35.0+k3s1
k3s-2   Ready    control-plane,etcd   9d    v1.35.0+k3s1
k3s-3   Ready    control-plane,etcd   9d    v1.35.0+k3s1

All these hosts are located in the same network.
The setup seems to work properly, except the "metrics".
The metrics port TCP/10250 is accesible via the internalIP (ip address of k3s host), but the K3S software is using the pod IP address.
The other two servers, that are not running the metrics-server pod, are not able to establish the TCP connection.

k3s[9613]: time="2026-01-22T13:25:40+01:00" level=error msg="Sending HTTP/1.1 502 response to 127.0.0.1:42134: dial tcp 10.42.1.34:10250: connect: connection timed out"
k3s[9613]: E0122 13:25:38.158801    9613 remote_available_controller.go:462] "Unhandled Error" err="v1beta1.metrics.k8s.io failed with: failing or missing response from https://10.42.1.34:10250/apis/metrics.k8s.io/v1beta1: Get \"https://10.42.1.34:10250/apis/metrics.k8s.io/v1beta1\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
 k3s[9613]: time="2026-01-22T13:25:36+01:00" level=error msg="Sending HTTP/1.1 502 response to 127.0.0.1:42104: dial tcp 10.42.1.34:10250: connect: connection timed out"
k3s[9613]: time="2026-01-22T13:25:36+01:00" level=error msg="Sending HTTP/1.1 502 response to 127.0.0.1:42118: dial tcp 10.42.1.34:10250: connect: connection timed out"

# kubectl top nodes
error: Metrics API not available
# kubectl top pods
error: Metrics API not available

In addition, all nodes are showing error messages about v1beta1.metrics.k8s.io, here an example:

E0122 15:16:55.784358   37037 remote_available_controller.go:462] "Unhandled Error" err="v1beta1.metrics.k8s.io failed with: Operation cannot be fulfilled on apiservices.apiregistration.k8s.io \"v1beta1.metrics.k8s.io\": the object has been modified; please apply your changes to the latest version and try again"

# kubectl get apiservice v1beta1.metrics.k8s.io -o yaml 
apiVersion: apiregistration.k8s.io/v1
kind: APIService
metadata:
  annotations:
    objectset.rio.cattle.io/applied: H4sIAAAAAAAA/4SSQY/aMBCF/0o1Z5NiQlKw...
    objectset.rio.cattle.io/id: ""
    objectset.rio.cattle.io/owner-gvk: k3s.cattle.io/v1, Kind=Addon
    objectset.rio.cattle.io/owner-name: metrics-apiservice
    objectset.rio.cattle.io/owner-namespace: kube-system
  creationTimestamp: "2026-01-12T14:50:50Z"
  labels:
    objectset.rio.cattle.io/hash: 54b5eb8b3ff563ca319415761629c9cbfaefe2a6
  name: v1beta1.metrics.k8s.io
  resourceVersion: "2258114"
  uid: 27f3ecf4-90a1-4042-90f1-26fad2e291d5
spec:
  group: metrics.k8s.io
  groupPriorityMinimum: 100
  insecureSkipTLSVerify: true
  service:
    name: metrics-server
    namespace: kube-system
    port: 443
  version: v1beta1
  versionPriority: 100
status:
  conditions:
  - lastTransitionTime: "2026-01-22T14:22:00Z"
    message: all checks passed
    reason: Passed
    status: "True"
    type: Available

Each node is able to establish TCP/10250 connection to another nodes using the remote host IP (10.10.10.5 -> 10.10.10.7).

[root@k3s-1]# nc -zv 10.10.10.5 10250 -> first node of k3s deployment
Ncat: Version 7.92 ( https://nmap.org/ncat )
Ncat: Connected to 10.10.10.5:10250.
Ncat: 0 bytes sent, 0 bytes received in 0.01 seconds.
[root@k3s-1]# nc -zv 10.10.10.6 10250 -> second node of k3s deployment
Ncat: Version 7.92 ( https://nmap.org/ncat )
Ncat: Connected to 10.10.10.6:10250.
Ncat: 0 bytes sent, 0 bytes received in 0.01 seconds.
[root@k3s-1]# nc -zv 10.10.10.7 10250 -> third node of k3s deployment
Ncat: Version 7.92 ( https://nmap.org/ncat )
Ncat: Connected to 10.10.10.7:10250.
Ncat: 0 bytes sent, 0 bytes received in 0.01 seconds.

The ip address "10.42.1.34" exists in my scenario, it is reacheable just via ICMP from the three nodes.

[root@k3s-2]# /usr/local/bin/kubectl get pods --all-namespaces -o wide | grep 10.42.1.34
kube-system   metrics-server-7f4dc5f569-nr6qj           1/1     Running            0              73m    10.42.1.34   k <none>           <none> 

In the end, I am able to see metrics from just one of this node using the command line "kubectl top".

How can I solve this issue?
Can I force to use just the internalIP?
Thanks :)

[brandond]

The metrics port TCP/10250 is accesible via the internalIP (ip address of k3s host), but the K3S software is using the pod IP address.

Yes, that is how it works. The metrics-server pods scrapes the nodes on the kubelet metrics port, and aggregates the data in memory. The apiserver proxies metrics requests to the metrics-server pod, which serves metrics.

In this case, it appears that the apiserver on the nodes not running the metrics-server pod cannot reach the pod by its IP. Are you sure you have the correct ports open between nodes? Check the port requirements doc, and make sure you don't have any conflicting firewall/iptables rules blocking traffic between nodes. If flannel is working, you should be able to curl the metrics-server pod (https://10.42.1.34:10250 in your logs) from any node in the cluster.

[fedesysadmin]

Hi @brandond ,
thanks for your hints!

Yes, the nodes not running the metrics-server are not able to reach the pod by its IP.
On my nodes there is a "flannel" interface, but I do not know if it is enough.
There is not a pod called "flannel" in my kube-system namespace

4: flannel.1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UNKNOWN group default 
    link/ether ea:17:15:d2:22:4f brd ff:ff:ff:ff:ff:ff
    inet 10.42.0.0/32 scope global flannel.1
       valid_lft forever preferred_lft forever
5: cni0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP group default qlen 1000
    link/ether 4a:ac:51:34:ec:46 brd ff:ff:ff:ff:ff:ff
    inet 10.42.0.1/24 brd 10.42.0.255 scope global cni0
       valid_lft forever preferred_lft forever

[root@k3s-1]# kubectl -n kube-system get pods
NAME                                      READY   STATUS    RESTARTS       AGE
coredns-54bf7cdff9-6btch                  1/1     Running   4 (11h ago)    3d13h
local-path-provisioner-69879d7dd7-5g9jm   1/1     Running   4 (11h ago)    3d13h
metrics-server-77dbbf84b-gcxmm            1/1     Running   0              56m
svclb-traefik-9e5874b6-khmrj              2/2     Running   14 (11h ago)   10d
svclb-traefik-9e5874b6-s9bhc              2/2     Running   12 (11h ago)   10d
svclb-traefik-9e5874b6-zbqtw              2/2     Running   18 (11h ago)   10d
traefik-6d98778dfc-trxxf                  1/1     Running   4 (11h ago)    3d13h

Have I to install this component manually?
Thanks

[brandond]

no. flannel is built in to the k3s binary. it does not run as a pod.

Check the K3s docs regarding firewall rules. Technically we don't support operating with ufw/firewalld enabled, but there is a port list in the docs.

[fedesysadmin]

Hi @brandond
thanks for your help and your hints!
I decided to inspect the network traffic using tcpdump and Wireshark to investigate further and determine the cause of the TCP timeout connections.
ICMP communication appeared to be working correctly, but Wireshark detected anomalies; some ICMP requests were not receiving a response (not visible from RHEL CLI).
The 10.42.0.89 IP Address that you see in the picture was the metrics-server IP Address during the troubleshooting.

I searched for more information about flannel and found the solution in this thread: #5013
I am running RHEL 8.10 (updated) and it is still affected by this known bug.

To solve the issue I decided to setup a SystemD services called "disable-flannel-checksum.service"

Creation and set up of the new service:

nano /etc/systemd/system/disable-flannel-checksum.service

[Unit]
Description=Disable tx-checksum-ip-generic on flannel.1
After=network-online.target k3s.service
Wants=network-online.target k3s.service

[Service]
Type=oneshot
ExecStart=/bin/bash -c "for i in {1..30}; do ip link show flannel.1 >/dev/null 2>&1 && /usr/sbin/ethtool -K flannel.1 tx-checksum-ip-generic off && exit 0; sleep 2; done; exit 1"
RemainAfterExit=yes

[Install]
WantedBy=multi-user.target

Activation of this new custom service

sudo systemctl daemon-reload
sudo systemctl enable disable-flannel-checksum.service
sudo systemctl start disable-flannel-checksum.service
sudo systemctl status disable-flannel-checksum.service

K3S nodes that are not running the metrics-server pod can reach the metrics-server pod via TCP/10250 as expected.

Thanks a lot
Federico