rootless dns resolv.conf race condition?

[flyingfishflash]

Am using pasta.

I'm trying to understand why my rootless containers, which start on boot via a series of quadlets, are getting an incorrect resolv.conf?

I am specifying a custom network, which is simply [Network] for each quadlet to use.

If I remove this Network= statement from the quadlet, I don't have this problem. But start to have port conflict issues as all quadlets now share the same network.

I am using systemd-resolved with stub-resolver.

This seems to be some sort of condition where podman or pasta is getting my dns configuration before it has properly been assigned via dhcp or router advertisement.

If after all the containers kicked off during boot have started, I run this while logged in as the same user under which all the containers are running:

podman run --rm alpine:latest sh -c "apk add --quiet --no-cache bind-tools && cat /etc/resolv.conf && time dig +tcp google.com"

The properly configured resolv.conf is output, that includes my LAN's local DNS ip address.

Why do none of the previously started containers have the proper resolv.conf.

Thinking this is some sort if timing issue, I modified podman-user-wait-network-online.service to just be a 1 minute delay.

(Perhaps whatever is configuring dns for these containers as they start will now be using the fully configured resolv.conf)

Yet my rootless containers started via quadlets with their custom networks have a resolv.conf that is from earlier in the boot process, not the one in /run/systemd/resolve/resolv.conf at the time they are starting. I verified this my modifying systemd-network-online.service to output my resolv.conf, and indeed it is not yet configured properly and the same content as what is showing in my containers even with a one minute delay.

Any suggestions on how get the proper resolv.conf loaded? I would rather not mount /run/system/resolve/resolv.conf into each container... but if I have to... I know I can also just remove the custom networks and juggle the ports around so there are no conflicts.

Is there another podman related service I can delay until my DNS configuration is as it should be?

-- Base on stat, creation time, my /run/systemd/resolve/resolv.conf is written here, after the container-runner use session in started, and before the delay ends and pods begin to startup:

Jul 15 16:28:50 nas sh[783]: Delaying Pod start... (9/45)
Jul 15 16:28:50 nas systemd-networkd[477]: br1: Gained IPv6LL
Jul 15 16:28:50 nas systemd-timesyncd[458]: Network configuration changed, trying to establish connection.
Jul 15 16:28:51 nas systemd-resolved[457]: Clock change detected. Flushing caches.
Jul 15 16:28:51 nas systemd-timesyncd[458]: Contacted time server 149.28.200.179:123 (2.arch.pool.ntp.org).
Jul 15 16:28:51 nas systemd-timesyncd[458]: Initial clock synchronization to Tue 2025-07-15 16:28:51.449294 EDT.
Jul 15 16:28:51 nas sh[783]: Delaying Pod start... (10/45)
Jul 15 16:28:52 nas sh[783]: Delaying Pod start... (11/45)
**Jul 15 16:28:52 nas systemd-networkd[477]: br1: DHCPv6 address fd99:9999:9999::9999/128 (valid for 5min 49s, preferred for 5min 49s)**
**Jul 15 16:28:52 nas systemd-networkd[477]: br1: DHCPv6 address [redacted]/128 (valid for 5min 49s, preferred for 5min 49s)**
Jul 15 16:28:53 nas sh[783]: Delaying Pod start... (12/45)
Jul 15 16:28:54 nas sh[783]: Delaying Pod start... (13/45)
Jul 15 16:28:55 nas sh[783]: Delaying Pod start... (14/45)
Jul 15 16:28:56 nas sh[783]: Delaying Pod start... (15/45)

Thanks,
-Greg

[flyingfishflash]

In case it helps anyone else.

1. I added ipv6 to each network quadlet

[Network]
DisableDNS=true
Driver=bridge
IPv6=true
Subnet=fdfd::/64

2. I ran systemctl --user daemon-reload and rebooted. Did not work, a new network was not generated, it was still using the previous one (I validated the uuid).

3. I deleted all the podman networks generated by systemd (podman network delete xxx).

4. lo and behold my containers now come up with the proper resolv.conf content.

[Luap99]

DisableDNS=true

Do you use that for all networks before as well? Just wondering because changing ipv6 should make no real difference in terms of timing or dns setup, except that that our logic also keeps ipv6 nameservers in the containers resolv.conf. So I guess it is possible you have a working ipv6 nameserver in your hosts resolv.conf then?

[Luap99]

There are in general a fair amount of race condition on startup since we just copy the hosts resolv.conf which of course would fail if it is not fully ready.

I modified podman-user-wait-network-online.service to just be a 1 minute delay.

I would assume that should generally work, are you sure the network was fully configured after 1 minute and resolv.conf had the right content?

One thing to note here (if DisableDNS=true is not used) dns will routed through aardvark-dns which listens on the bridge ip. And aardvark-dns will read the custom resolv.conf from the rootless-netns. podman unshare --rootless-netns cat /etc/resolv.conf should show it. The rootless-netns is setup when the first container is started so the resolv.conf will generally stay the same there so if it starts to early even restarting a single container would not help when using aardvark-dns.
See #22943 (comment) on how the rootless-netns works.

[flyingfishflash]

I have continued looking into this, and it seems to be some kind of routing issue possibly on my side. The dns is working properly whether DisableDNS is True or False. Content of resolv.conf is different, but resolution is still happening.

I do have a working DNS server on my LAN (Opnsense -> Unbound -> DnsMasq).

The issue is accessing anything via ipv6 outside the container. So perhaps I am missing an additional route that needs to be configured on my router or elsewhere.

 podman exec -it tautulli-app curl -v -6 -k https://google.com
*   Trying [2607:f8b0:4004:c1f::71]:443...
* connect to 2607:f8b0:4004:c1f::71 port 443 failed: Network is unreachable
*   Trying [2607:f8b0:4004:c1f::8a]:443...
* connect to 2607:f8b0:4004:c1f::8a port 443 failed: Network is unreachable

created via quadlet

[
     {
          "name": "systemd-tautulli",
          "id": "27f663bd9ed28fa4ea5ef6e57dbe002341bda6b3f76be3c3bfcd6d3f096d7035",
          "driver": "bridge",
          "network_interface": "podman16",
          "created": "2025-07-16T09:05:02.190445507-04:00",
          "subnets": [
               {
                    "subnet": "10.89.15.0/24",
                    "gateway": "10.89.15.1"
               },
               {
                    "subnet": "fd17:2df2:386:614f::/64",
                    "gateway": "fd17:2df2:386:614f::1"
               }
          ],
          "ipv6_enabled": true,
          "internal": false,
          "dns_enabled": true,
          "ipam_options": {
               "driver": "host-local"
          },
          "containers": {
               "7e815af6bb8101fb2ee056a039a8f1a4bab419f0622d37bcf2d541617ea734fb": {
                    "name": "tautulli-app",
                    "interfaces": {
                         "eth0": {
                              "subnets": [
                                   {
                                        "ipnet": "10.89.15.2/24",
                                        "gateway": "10.89.15.1"
                                   },
                                   {
                                        "ipnet": "fd17:2df2:386:614f::2/64",
                                        "gateway": "fd17:2df2:386:614f::1"
                                   }
                              ],
                              "mac_address": "7a:0f:4c:e0:d8:02"
                         }
                    }
               }
          }
     }
]