DNS resolving not working in ephemeral guests

[chrboe]

Guests started by bcvk ephemeral are unable to resolve hostnames, even though internet access is available.

To reproduce:

$ bcvk ephemeral run-ssh quay.io/fedora/fedora-bootc:42 
[root@fedora ~]# ping 1.1.1.1
PING 1.1.1.1 (1.1.1.1) 56(84) bytes of data.
64 bytes from 1.1.1.1: icmp_seq=1 ttl=255 time=3.54 ms
64 bytes from 1.1.1.1: icmp_seq=2 ttl=255 time=3.32 ms
^C
--- 1.1.1.1 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1002ms
rtt min/avg/max/mdev = 3.315/3.429/3.543/0.114 ms
[root@fedora ~]# systemd-resolve google.com
google.com: resolve call failed: Lookup failed due to system error: Network is unreachable

It seems that the DNS server is configured to 10.0.2.3:

[root@fedora ~]# resolvectl status
Global
         Protocols: LLMNR=resolve -mDNS -DNSOverTLS DNSSEC=no/unsupported
  resolv.conf mode: stub

Link 2 (ens5)
    Current Scopes: DNS LLMNR/IPv4 LLMNR/IPv6
         Protocols: +DefaultRoute LLMNR=resolve -mDNS -DNSOverTLS DNSSEC=no/unsupported
Current DNS Server: 10.0.2.3
       DNS Servers: 10.0.2.3
     Default Route: yes

I can ping this IP address, but I am unsure where exactly it comes from or how to further debug this.

[cgwalters]

Yeah there's two levels of emulated networking, first the default podman run allocates a bridge device, and then we have qemu's userspace networking. I haven't dug into this but the first DNS server here is from qemu.

# DNS Resolution Failure in Ephemeral VMs - Issue #123

Problem Statement

DNS resolution fails in ephemeral guests started by bcvk ephemeral run. While IP connectivity works (can ping 1.1.1.1), hostname resolution fails (systemd-resolve google.com times out).

Symptoms

# In guest VM
$ ping 1.1.1.1  # Works
$ ping google.com  # Fails - unknown host
$ resolvectl status
# Shows DNS server: 10.0.2.3 (QEMU's slirp DNS forwarder)
$ ping 10.0.2.3  # Works - slirp is reachable
$ nslookup google.com 10.0.2.3  # Times out - DNS queries fail

Root Cause Analysis

Architecture Context

Ephemeral VMs run through this chain:

1. Host → Podman container (privileged, with container image mounted)
2. Container → QEMU process with user-mode networking (slirp)
3. QEMU → Guest VM with VirtioFS root filesystem

The Problem

QEMU's user-mode networking (slirp) provides a built-in DNS forwarder at 10.0.2.3. This forwarder:

1. Reads /etc/resolv.conf from its environment (the container)
2. Forwards guest DNS queries to those nameservers

Container's /etc/resolv.conf contains:

nameserver 169.254.1.1     # Container bridge network (podman's dnsname plugin)
nameserver 10.11.5.19       # Container network DNS
nameserver 10.2.32.1        # Container network DNS

The critical issue:

• These DNS servers are only accessible within the container's network namespace
• QEMU's slirp runs in an isolated user-mode network
• Slirp can't reach 169.254.1.1 or the 10.x.y.z container bridge IPs
• DNS queries are forwarded to unreachable servers → timeout

Why This Is Different From Running QEMU Directly on Host

When QEMU runs directly on the host:

• Slirp reads host's /etc/resolv.conf
• Host typically has nameserver 127.0.0.53 (systemd-resolved)
• systemd-resolved forwards to real DNS servers (8.8.8.8, ISP DNS, etc.)
• Everything works

When QEMU runs in a podman container:

• Slirp reads container's /etc/resolv.conf
• Container has bridge network DNS servers
• Those servers are in a different network namespace
• DNS forwarding fails

Solution Attempt 1: --net=host

Implementation

Modified /var/home/walters/src/github/bootc-dev/bcvk/crates/kit/src/run_ephemeral.rs:

// Use host networking by default for ephemeral VMs. This avoids DNS resolution
// issues that occur when QEMU's slirp tries to use container bridge DNS servers
// (e.g., 169.254.1.1 or 10.x.y.z) which are unreachable from QEMU's isolated network.
// Port forwarding is handled at the QEMU level (guest -> host).
let network = opts.podman.network.as_deref().unwrap_or("host");
cmd.args(["--network", network]);

How This Fixes DNS

With --net=host:

• Container shares host's network namespace
• QEMU's slirp reads host's /etc/resolv.conf
• Gets nameserver 127.0.0.53 (systemd-resolved on host)
• DNS queries work because host's DNS infrastructure is accessible

New Problem: Port Conflicts

The Issue:

• With --net=host, QEMU's SSH port forwarding (tcp::2222-:22) binds to host's port 2222
• Multiple VMs now conflict on the same port
• bcvk ephemeral ssh hardcodes port 2222 and fails

Example:

$ bcvk ephemeral run --name vm1 -K -d image:latest  # Binds to host:2222
$ bcvk ephemeral run --name vm2 -K -d image:latest  # FAILS - port 2222 already in use
$ bcvk ephemeral ssh vm1  # Tries to connect to localhost:2222

Solution Considerations

Option 1: Dynamic Port Allocation

• Allocate random/sequential ports for each VM
• Store port mapping somewhere (container labels? state file?)
• bcvk ephemeral ssh queries the mapping to find correct port
• Challenge: Race conditions, port tracking complexity

Option 2: Port Namespacing by Container Name

• Use deterministic port based on container name hash
• E.g., hash(container_name) % 10000 + 10000 → port 10000-19999 range
• Still risk of collisions, but predictable
• Challenge: Hash collisions, hard to remember ports

Option 3: Per-VM Vsock Instead of TCP

• Use AF_VSOCK for guest-host communication
• Each VM gets unique CID (Context ID)
• No port conflicts possible
• Challenge: More complex implementation, vsock support required

Option 4: Back to Bridge Networking + Passt

• Use passt/pasta for container networking instead of slirp
• Passt is designed for containerized environments
• Modern replacement for slirp with better DNS handling
• Benefit: Solves DNS properly without --net=host
• Challenge: More complex integration, new dependency

Similar Issue in Libvirt Flow

The comment notes: "this is similar for libvirt" - the libvirt integration likely has similar port allocation challenges. Solving this for ephemeral should inform the libvirt approach.

Testing Performed

Reproducer (Original Issue)

$ bcvk ephemeral run --name test-net -K -d quay.io/fedora/fedora-bootc:42
$ podman exec test-net cat /etc/resolv.conf
nameserver 169.254.1.1
nameserver 10.11.5.19
nameserver 10.2.32.1
$ bcvk ephemeral ssh test-net resolvectl status
# Shows DNS: 10.0.2.3
$ bcvk ephemeral ssh test-net ping 10.0.2.3
# Works
$ bcvk ephemeral ssh test-net nslookup google.com
# Timeout - confirms bug

With --net=host Fix

$ cargo build --release
$ ./target/release/bcvk ephemeral run --name test-net2 -K -d quay.io/fedora/fedora-bootc:42
# Container starts with --network=host
$ ./target/release/bcvk ephemeral ssh test-net2 ping google.com
# Timeout - SSH can't connect (port conflict or VM not ready)

Next Steps

1. Decide on port allocation strategy - likely dynamic allocation with state tracking
2. Implement port tracking - container labels or supervisor state file
3. Update bcvk ephemeral ssh - query port from container metadata
4. Consider passt integration - may be cleaner long-term solution
5. Apply solution to libvirt flow - similar challenges exist there

References

• Issue: DNS resolving not working in ephemeral guests #123
• QEMU user networking: https://wiki.qemu.org/Documentation/Networking#User_Networking_(SLIRP)
• Passt/Pasta: https://passt.top/passt/about/
• Libvirt passt: https://libvirt.org/formatdomain.html#userspace-connection-using-passt

[cgwalters]

I was thinking about this more recently, and bigger picture what would likely be helpful here is to rework things so that bcvk ephemeral is actually just "libvirt-in-container" - contrasting with bcvk libvirt being libvirt-outside-of-container.

The connection here is, I think we always want libvirt to be in charge of networking, which argues for fixing the --net=host approach.

We already need to deal with port conflicts on the bcvk libvirt side anyways, so doing it for ephemeral isn't really any different.

[gursewak1997]

Based on the above comment about treating bcvk-ephemeral as “libvirt-in-container” and always letting libvirt own networking, I updated the implementation to use libvirt’s NAT network instead of slirp. This resolves the DNS issues I was hitting earlier.
This design would require libvirtd to be running inside the container, which is a heavier dependency than slirp. I assume this is acceptable, but I want to verify before finalizing the behavior.

Would you prefer:
Hard-require libvirtd for bcvk-ephemeral (no fallback), or
Add a fallback to slirp for environments where libvirt isn’t available (with the known DNS limitations)?

[cgwalters]

The core way that bcvk ephemeral works today is actually to run the whole host system as a container; it's not too much more of a lift to run libvirt than it is directly qemu. Now a wrinkle here is that I think qemu:///session is kind of designed to run as part of a login session (which a container won't feel like by default) but it's likely surmountable.

[cgwalters]

We currently hard require libvirt for the bcvk libvirt verb. That said I would like to support e.g. systemd-vmspawn too in the future, so maybe it becomes optional.

Either way I think the idea here is bcvk ephemeral switches over to looking more like a "local production" VM path just with the container runtime involved to mount the rootfs.