Failed to execute portforward (failed to connect to localhost:80 inside namespace)

[baflo]

Environmental Info:
K3s Version:
k3s version v1.23.6+k3s1 (418c3fa)
go version go1.17.5

Node(s) CPU architecture, OS, and Version:
Linux arch-3-k3s 5.18.1-arch1-1 #1 SMP PREEMPT_DYNAMIC Mon, 30 May 2022 17:53:11 +0000 x86_64 GNU/Linux

Cluster Configuration:
Single node setup in rootless mode

Describe the bug:

When acceessing a service through kubectl port-forward the request is declined with the following error message:

$ kubectl port-forward nginx 8080:80
Forwarding from 127.0.0.1:8080 -> 80
Forwarding from [::1]:8080 -> 80
Handling connection for 8080
E0627 07:30:06.723465   12799 portforward.go:400] an error occurred forwarding 8080 -> 80: error forwarding port 80 to pod 1b70fbf0c85a96e02240f03eab98e844e121bece4f2fae7f523c6be19ce33c15, uid : failed to execute portforward in network namespace "/var/run/netns/cni-a3773a81-246d-5f36-3c45-fc0b9e117e5d": failed to connect to localhost:80 inside namespace "1b70fbf0c85a96e02240f03eab98e844e121bece4f2fae7f523c6be19ce33c15", IPv4: dial tcp4: lookup localhost on 10.41.0.3:53: no such host IPv6 dial tcp6: lookup localhost on 10.41.0.3:53: no such host

The host 10.41.0.3 actually does not exist, at least not as pod or service:

$ kubectl get svc,pods -Aowide
NAMESPACE     NAME                     TYPE           CLUSTER-IP      EXTERNAL-IP   PORT(S)                      AGE   SELECTOR
default       service/kubernetes       ClusterIP      10.43.0.1       <none>        443/TCP                      14m   <none>
kube-system   service/kube-dns         ClusterIP      10.43.0.10      <none>        53/UDP,53/TCP,9153/TCP       14m   k8s-app=kube-dns
kube-system   service/metrics-server   ClusterIP      10.43.13.29     <none>        443/TCP                      13m   k8s-app=metrics-server
kube-system   service/traefik          LoadBalancer   10.43.248.255   127.0.0.1     80:32534/TCP,443:30647/TCP   13m   app.kubernetes.io/instance=traefik,app.kubernetes.io/name=traefik

NAMESPACE     NAME                                          READY   STATUS      RESTARTS   AGE   IP          NODE         NOMINATED NODE   READINESS GATES
kube-system   pod/coredns-d76bd69b-k7955                    1/1     Running     0          13m   10.42.0.4   arch-3-k3s   <none>           <none>
kube-system   pod/local-path-provisioner-6c79684f77-qfk6l   1/1     Running     0          13m   10.42.0.5   arch-3-k3s   <none>           <none>
kube-system   pod/helm-install-traefik-crd-q6q25            0/1     Completed   0          13m   10.42.0.3   arch-3-k3s   <none>           <none>
kube-system   pod/helm-install-traefik-q267k                0/1     Completed   1          13m   10.42.0.2   arch-3-k3s   <none>           <none>
kube-system   pod/svclb-traefik-6zwwk                       2/2     Running     0          13m   10.42.0.7   arch-3-k3s   <none>           <none>
kube-system   pod/metrics-server-7cd5fcb6b7-7rhw6           1/1     Running     0          13m   10.42.0.6   arch-3-k3s   <none>           <none>
kube-system   pod/traefik-df4ff85d6-8pznc                   1/1     Running     0          13m   10.42.0.8   arch-3-k3s   <none>           <none>
default       pod/nginx                                     1/1     Running     0          11m   10.42.0.9   arch-3-k3s   <none>           <none>

Steps To Reproduce:

• Installed current Arch Linux on Hyper-V VM

• Installed fuse-overlayfs

• Set up file /etc/systemd/system/user@.service.d/delegate.conf

  [Service]
  Delegate=cpu cpuset io memory pids
  

• Set up subordinate files

  $ cat /etc/subuid
  fb:100000:65536
  du:165536:65536
  $ cat /etc/subgid
  fb:100000:65536
  du:165536:65536
  

• Installed K3s:

  • as described in https://raw.githubusercontent.com/k3s-io/k3s/v1.23.6%2Bk3s1/k3s-rootless.service

• Created nginx pod on k3s: kubectl run --image=nginx nginx

• Run kubectl port-forward nginx 8080:80

• Then run curl localhost:8080

Expected behavior:
The port forward should work in rootless mode.

Actual behavior:
The port forward does not work in rootless mode.

Additional context / logs:

Backporting

• Needs backporting to older releases

[brandond]

Hmm these errors look pretty weird. Any ideas @AkihiroSuda ?

IPv4: dial tcp4: lookup localhost on 10.41.0.3:53: no such host 
IPv6 dial tcp6: lookup localhost on 10.41.0.3:53: no such host

[baflo]

@AkihiroSuda @brandond Just tried the same machine with rooted k3s. I disable rootless k3s and installed "rootful" following this guide: https://rancher.com/docs/k3s/latest/en/installation/install-options/

Turned out I still have the same problem.

[AkihiroSuda]

@AkihiroSuda @brandond Just tried the same machine with rooted k3s. I disable rootless k3s and installed "rootful" following this guide: https://rancher.com/docs/k3s/latest/en/installation/install-options/

Turned out I still have the same problem.

Does it work if you disable firewalld or something similar?

[elek]

I had similar problem with archlinux + rootful k3s, and fixed by adding 127.0.0.1 localhost to my /etc/hosts (localhost name was missing for 127.0.0.1).

[beanaroo]

I have been getting a similar error (fresh install of Arch Linux + k3s):

IPv4: dial tcp4: lookup localhost on 1.1.1.1:53: no such host 
IPv6 dial tcp6: lookup localhost on 1.1.1.1:53: no such host

Adding 127.0.0.1 localhost to /etc/hosts as suggested above resolved it.

[shankarm123]

in POD deployment yaml you need to set JAVA_DEBUG=true