dns resolution not working from host

[tcurdt]

For some reason dns resolution is not working and I don't see the problem.

This creates the pod.

podman network create live-echo
podman pod create live-echo
podman run --pod live-echo --network live-echo --name live-frontend -dt docker.io/traefik/whoami:v1.9.0

Now I want resolve the IP from a container. I am not sure --network works on a container.

podman run --network live-echo --rm alpine cat /etc/hosts
podman run --network live-echo --rm alpine nslookup live-frontend # fails

But it should work on a pod

podman run --pod live-echo --network live-echo --rm alpine cat /etc/hosts
podman run --pod live-echo --network live-echo --rm alpine nslookup live-frontend # fails

Now I am digging a little deeper. Inspecting the container I can get the IP and reach the container:

$ docker inspect live-frontend | grep IPAddress
"IPAddress": "10.89.1.23",
$ curl 10.89.1.23
Hostname: live-echo

This is what is in the hosts file.

$ podman run --pod live-echo --network live-echo --rm alpine cat /etc/hosts
127.0.0.1    localhost
::1    localhost
127.0.0.2    nixos.utm nixos
::1    nixos.utm nixos
10.89.1.1    host.containers.internal host.docker.internal
10.89.1.24    live-echo relaxed_banach

Now checking the resolv.conf

$ podman run --pod live-echo --network live-echo --rm alpine cat /etc/resolv.conf
search dns.podman
nameserver 10.89.1.1
options edns0

Trying the lookup from the server directly from the host works:

$ nslookup
> server 10.89.1.1
Default server: 10.89.1.1
Address: 10.89.1.1#53
> live-frontend
Server:        10.89.1.1
Address:    10.89.1.1#53

Non-authoritative answer:
Name:    live-frontend
Address: 10.89.1.23

But for some reason the lookup fails from the container:

$ podman run --pod live-echo --network live-echo --rm alpine nslookup live-frontend
;; connection timed out; no servers could be reached

Why?

[rhatdan]

@Luap99 PTAL

[Luap99]

What podman version is this? Are you using netavark/aardvark-dns? Are you running rootless?

Also you should only set --network on the pod create command, not the the individual containers that are joined to this pod. While it is a valid configuration it is likely not what you want. A pod is used so that all container share the network. If you give each container a new --network then they will not share the network and instead have their own separate network stack.

[tcurdt]

That's podman version 4.7.2. AFAIU it should not be running rootless.
Here is my nixos flake. If you want you could reproduce the full system.
Here is the podman configuration in nixos.

Here are a few more finding.
When I only assign the network to the pod things do look a little different.

$ docker inspect live-frontend | grep IPAddress
"IPAddress": "10.89.1.15",

$ curl 10.89.1.15
Hostname: live-echo
IP: 127.0.0.1
IP: 10.89.1.15


$ podman run --pod live-echo --rm alpine cat /etc/hosts
127.0.0.1	localhost
::1	localhost
127.0.0.2	nixos.utm nixos
::1	nixos.utm nixos
10.89.1.1	host.containers.internal host.docker.internal
10.89.1.15	live-echo 13e538aa6ed1-infra
127.0.0.1	live-frontend
127.0.0.1	confident_jang

$ podman run --pod live-echo --rm alpine cat /etc/resolv.conf
search dns.podman
nameserver 10.89.1.1

$podman run --pod live-echo --rm alpine nslookup live-frontend
;; connection timed out; no servers could be reached

I am wondering whether it is just the lookup but a ping fails - not sure why.

podman run --pod live-echo -u root --rm alpine ping live-frontend
PING live-frontend (127.0.0.1): 56 data bytes
ping: permission denied (are you root?)

What does work now:

podman run --pod live-echo --rm -it quay.io/curl/curl:latest curl live-frontend

🎉 That's a first win.

Now a container that has access to both networks feels kind of necessary but does not work

podman run --network live-echo --network test-echo -it quay.io/curl/curl:latest curl live-frontend
curl: (6) Could not resolve host: live-frontend

Tapping into two pods (not even sure this is a valid idea - but it does not complain) also fails

podman run --pod live-echo --pod test-echo -it quay.io/curl/curl:latest curl live-frontend
curl: (6) Could not resolve host: live-frontend

I am looking for two things:

1. How could an "ingress container" reach both pods?
2. How can I use a reverse proxy that is running on the host?

I would need to be able to resolve and reach live-frontend in the host for (2).

As shown above, reaching the container seems no problem - but I need to resolve them.
And it seems there is no single root dns resolver I can use.
They seem to be dynamic depending on the network/pod existence.

ss -ntulp | grep aardvark-dns
udp   UNCONN 0      0          10.89.0.1:53        0.0.0.0:*    users:(("aardvark-dns",pid=6916,fd=13))   
udp   UNCONN 0      0          10.89.1.1:53        0.0.0.0:*    users:(("aardvark-dns",pid=6916,fd=12))

Which is a problem when I would want to add them to the host resolv.conf.

Am I trying to do something weird here?
Seems like such a normal case to expose the pods.

[tcurdt]

Any further suggestions? I am quite stuck here.

[Luap99]

Sorry I do not have time look into this, if you are using a pod try to resolve the pod name not the container name.

[tcurdt]

Even more thanks for responding anyway. It did help a bit to understand the situation better.
I keep the testcase here for easy replication and reference:

podman network create foo
podman pod create --network foo foo
podman run --pod foo --name foo-frontend -dt docker.io/traefik/whoami:v1.9.0

podman network create bar
podman pod create --network bar bar
podman run --pod bar --name bar-frontend -dt docker.io/traefik/whoami:v1.9.0

podman run --pod foo --rm alpine cat /etc/hosts
10.89.0.2 foo 0e3d1b504ad3-infra
127.0.0.1 foo-frontend
podman run --pod foo --rm alpine nslookup foo # FAILS
podman run --pod foo --rm -it quay.io/curl/curl:latest curl foo # OK

podman run --pod bar --rm alpine cat /etc/hosts
10.89.1.2 bar 5e9118f60145-infra
127.0.0.1 bar-frontend
podman run --pod bar --rm alpine nslookup bar # FAILS
podman run --pod bar --rm -it quay.io/curl/curl:latest curl bar # OK

podman run --pod foo --pod bar --rm -it quay.io/curl/curl:latest curl bar # OK
podman run --pod foo --pod bar --rm -it quay.io/curl/curl:latest curl foo # FAILS
podman run --pod bar --pod foo --rm -it quay.io/curl/curl:latest curl bar # FAILS
podman run --pod bar --pod foo --rm -it quay.io/curl/curl:latest curl foo # OK

podman run --network bar --network foo --rm -it quay.io/curl/curl:latest curl foo # FAILS
podman run --network foo --network bar --rm -it quay.io/curl/curl:latest curl foo # FAILS
podman run --network bar --network foo --rm -it quay.io/curl/curl:latest curl bar # FAILS
podman run --network foo --network bar --rm -it quay.io/curl/curl:latest curl bar # FAILS

cat /etc/resolv.conf
nameserver 192.168.69.1

ping foo # FAILS
ping bar # FAILS
nslookup foo # FAILS
nslookup bar # FAILS
ping foo-frontend # FAILS
ping bar-frontend # FAILS
nslookup foo-frontend # FAILS
nslookup bar-frontend # FAILS

ss -ntulp | grep aardvark-dns
10.89.1.1:53        0.0.0.0:*    users:(("aardvark-dns",pid=2617,fd=13))
10.89.0.1:53        0.0.0.0:*    users:(("aardvark-dns",pid=2617,fd=12))

nslookup bar 10.89.1.1 # OK
nslookup foo 10.89.1.1 # FAILS
nslookup bar-frontend 10.89.1.1 # FAILS
nslookup foo-frontend 10.89.1.1 # FAILS

nslookup bar 10.89.0.1 # OK
nslookup foo 10.89.0.1 # FAILS
nslookup bar-frontend 10.89.0.1 # FAILS
nslookup foo-frontend 10.89.0.1 # FAILS

docker inspect foo-frontend | grep IPAddress
     "IPAddress": "10.89.0.2",
curl 10.89.0.2 # OK

docker inspect bar-frontend | grep IPAddress
     "IPAddress": "10.89.1.2",
curl 10.89.1.2 # OK

There are a couple of things that I think are worth exploring:

1. How can aardvark-dns instances be added to the host resolv.conf?
2. Why is the container IP (foo-frontend and bar-frontend) not resolved by the aardvark-dns instance?

[jammus]

@tcurdt - I was having similar problems. Short story is I set defaultNetwork.settings.dns_enabled = false;, then deleted and recreated the pod and it worked

Slightly longer story (which might not be 100% accurate because there was a lot of retrying and also a nix flake update in the middle)...

The test case podman run --pod lookup-test --rm -it busybox busybox nslookup nixos.org would work but sudo podman run --pod lookup-test --rm -it busybox busybox nslookup nixos.org would not. I had created two pods named lookup-test, one with sudo and one without. I then inspected both and diffed the output. That showed me that the sudo created one was using the podman bridge. I then looked and noticed there were two podman bridges with different ids depending on if I ran podman network ls with or without sudo. Diffing those the only different was the root accessible one was the dns setting. So I switched the setting and it worked.

Not really sure what's going on or if I'll break other things... but there you go. Maybe it'll work for you

[tcurdt]

Will give that a try. At least I am sure now I am not alone seeing the problem.

[tcurdt]

Alright this put me on the right track!

There is a ticket open #20562 (comment)

It seems like others have similar problems. Whether it works or not seem to depend whether dns is enabled or disabled on the network for the pod. But that seems just like a workaround. (And I wouldn't even know how to select the network or how to disable the internal dns via kube play manifest.)

The real question is why the dns resolver is not reachable.

# podman pod ls
POD ID        NAME        STATUS      CREATED         INFRA ID      # OF CONTAINERS
f70e047f31df  live        Running     19 seconds ago  772ffe41e03a  3

# podman exec -it live-shell1 sh
/ # nslookup live
;; connection timed out; no servers could be reached

This does feel like bug to me, @Luap99

Is there anything else I can do to help track this down?
Should I open another issue?

[tcurdt]

This seems to be a NixOS issue.
The package does not open the firewall correctly by default.
A fix is being discussed.

[sandangel]

Hello, I'm using fedora and have the same issue.