systemd-resolved and aardvark-dns with disabled stub-resolver

[rheaalleen]

Hey,

I was deploying PowerDNS to Podman containers and did encounter some difficulties.

PowerDNS container: https://hub.docker.com/r/pschiffe/pdns-pgsql/
PostgreSQL container: https://hub.docker.com/_/postgres/

Iḿ running Debian 12 which has systemd-resolved enabled and has the stub-listener which occupies Port 53. With my playbooks the containers should use these ports so I disabled the stub in /etc/systemd/resolved.conf.d/disable-stub.conf with

[Resolve]
DNSStubListener=no

Additionally in /etc/containers/containers.conf I specified the network_backend

[network]
network_backend = "netavark"

The setup with the container was that they are in an internal network and the database is available through the container name without an open port on the host.

When I restart the systemd-resolved service the internal networking breaks, the dns containers cant reach the database over container names anymore, why?

Is the internal dns resolution tied to systemd-resolved + aardvark-dns?

How can I keep the internal network working so that the containers can talk to each other with their names when I disable the stub-resolver? While troubleshooting I've also found that while all containers are in an internal network the database containers can still resolve external domains (when systemd-resolve was running, i.e. google.com) but the dns containers couldnt.

Only difference was that the db/dns containers had different resolv.confs and that DB container is based on alpine and DNS container is Fedora

	DNS	DB
search	dns.podman domain.io	domain.io
nameserver	10.89.0.1	10.89.0.1
nameserver	10.0.104.1	10.0.104.1

With enabled systemd-resolved but disabled stub and with/without specifying internal network the DB/DNS containers will resolve the container names when pinging to the WAN IP of the VPS.

[Luap99]

When I restart the systemd-resolved service the internal networking breaks, the dns containers cant reach the database over container names anymore, why?

aardvark-dns only reads resolv.conf on each container change (i.e. start/stop) and it currently does not have an active listener for resolv.conf changes. That also assume you use a recent version (debian 12 uses outdated stuff if you use official packages)

Anyway it needs to be fixed containers/aardvark-dns#485 (first point)

[rheaalleen]

The playbook forces the deletion of the container and services after the changes to systemd-resolved and the stub

Currently systemd-resolve is running but without stub-resolver but the resolv.conf stays the same? I dont see how the stub and resolv.conf interact with each other to provide container name resolution as normal DNS resolution works fine.

stub disabled

[root@ef8d032ebc7c /]# nslookup powerdns_admin
;; communications error to 10.89.0.1#53: connection refused
;; communications error to 10.89.0.1#53: connection refused
;; communications error to 10.89.0.1#53: connection refused
;; communications error to 10.89.0.1#53: connection refused
Server:         10.0.104.1
Address:        10.0.104.1#53

Non-authoritative answer:
Name:   powerdns_admin.domain.io
Address: XX.XX.XX.XX

[root@ef8d032ebc7c /]# ping powerdns_admin
PING powerdns_admin.domain.io (XX.XX.XX.XX) 56(84) bytes of data.

stub enabled

[root@a9a19b54bb6a /]# nslookup powerdns_admin
Server:         10.89.0.1
Address:        10.89.0.1#53

Non-authoritative answer:
Name:   powerdns_admin.dns.podman
Address: 10.89.0.11

[root@a9a19b54bb6a /]# ping powerdns_admin
PING powerdns_admin.dns.podman (10.89.0.11) 56(84) bytes of data.
64 bytes from 10.89.0.11: icmp_seq=1 ttl=64 time=0.036 ms
64 bytes from 10.89.0.11: icmp_seq=2 ttl=64 time=0.037 ms

The dns.podman entry is also not in the resolv.conf of the db containers, is that because of the different distro? Doesnt matter if stub is active or not, resolution of container names is not working, external working fine

[Luap99]

Sorry your setup is not clear to me, generally it is best to the with the latest versions.

If you bind 53 on the host that will cause conflicts with aardvark-dns binding the port. New versions should produce a clear error in such cases, if that is the case maybe this helps: #23740

[rheaalleen]

Podman is indeed a bit outdated in the version Debian 12 uses, it's Podman 4.3.1

As for the setup:

Out of the box the Debian systemd-resolved has the stub-resolver running

State                      Recv-Q                     Send-Q                                         Local Address:Port                                           Peer Address:Port                     Process                                                        
LISTEN                     0                          4096                                                 0.0.0.0:5355                                                0.0.0.0:*                         users:(("systemd-resolve",pid=386,fd=12))                     
LISTEN                     0                          4096                                              127.0.0.54:53                                                  0.0.0.0:*                         users:(("systemd-resolve",pid=386,fd=21))                     
LISTEN                     0                          4096                                           127.0.0.53%lo:53                                                  0.0.0.0:*                         users:(("systemd-resolve",pid=386,fd=19)) 

When I start the containers I cant use the port 53 since its in use, thats why (for testing) I use another

    ports:
      - "54:53"
      - "54:53/udp"

I exec into the containers and can use nslookup/ping successfully on container names. aardvark-dns is up and runnning on 10.89.0.1#53

Then I disable the stub-resolver to free up port 53, recreate the containers with the ports

    ports:
      - "53:53"
      - "53:53/udp"

After exec'ing into the container aardvark-dns refuses the connection as seen in my previous comment.

I've seen your comments in #23740 (comment), #23226, #23128 and #23740 but have yet to understand the interworking of aardvark, rootless and bindng it to the host ports.

I can confirm it works now after bindind it to host ip instead of 0.0.0.0