Restarting socket unit fails when other podman containers are running.

[michaelmedellin]

Upfront, socket activation is new to me but I'm experimenting with it for my caddy proxy to other containers running in podman.

I am using a nearly identical configuration to example 4 for testing, but I noticed that if I stop caddy.service and restart the socket systemctl --user restart caddy.socket it fails if other containers in podman are still running. It looks like aardvark and slirp4netns somehow end up binding to port 80 and the socket cannot bind on restart.

Obviously, don't plan to restart a socket often. But I noticed the issue and want to better understand if this expected behavior with sockets.

[eriksjolund]

Sorry for late reply. Do you have a step-by-step reproducer?

Are you using a recent Podman version?

Pasta has replaced slirp4netns as being the default since Podman 5.0.

I'm using Fedora 41 which contains recent versions of Podman and Pasta.

$ rpm -qf /usr/bin/pasta
passt-0^20250320.g32f6212-2.fc41.x86_64
$ rpm -qf /usr/bin/podman 
podman-5.4.1-1.fc41.x86_64
$ rpm -q aardvark-dns
aardvark-dns-1.14.0-1.fc41.x86_64
$ rpm -q netavark
netavark-1.14.0-1.fc41.x86_64

(The pasta version was released just 3 days ago!)

[bjorne]

I am running into the same issue. Reproduced on Debian trixie and Arch. I think the key is to have caddy part of a podman network that outlives the container, in which case aardvark-dns seems to cling onto the socket port for some reason.

Here are my repro steps (sorry for not basing this on your setup, which would probably have been easier in hindsight):

$ ss -tulpn | grep 8080                     
$ systemctl --user start test-main test-hello
$ systemctl --user restart test-main         
A dependency job for test-main.service failed. See 'journalctl -xe' for details.
$ journalctl --user -n30 --no-pager
[...]
Dec 31 13:54:27 dirac systemd[2125]: test-main.service: Main process exited, code=exited, status=1/FAILURE
Dec 31 13:54:27 dirac systemd[2125]: test-main.service: Failed with result 'exit-code'.
Dec 31 13:54:27 dirac systemd[2125]: test-main.service: Scheduled restart job, restart counter is at 1.
Dec 31 13:54:27 dirac systemd[2125]: Stopped Main container with socket activation.
Dec 31 13:54:27 dirac systemd[2125]: Closed Test HTTP Socket.
Dec 31 13:54:27 dirac systemd[2125]: Stopping Test HTTP Socket...
Dec 31 13:54:27 dirac systemd[2125]: test-http.socket: Failed to create listening socket ([::]:8080): Address already in use
Dec 31 13:54:27 dirac systemd[2125]: test-http.socket: Failed to listen on sockets: Address already in use
Dec 31 13:54:27 dirac systemd[2125]: test-http.socket: Failed with result 'resources'.
Dec 31 13:54:27 dirac systemd[2125]: Failed to listen on Test HTTP Socket.
Dec 31 13:54:27 dirac systemd[2125]: Dependency failed for Main container with socket activation.
Dec 31 13:54:27 dirac systemd[2125]: test-main.service: Job test-main.service/start failed with result 'dependency'.
$ podman ps                        
CONTAINER ID  IMAGE                            COMMAND     CREATED         STATUS         PORTS       NAMES
c3f2e6915d44  docker.io/traefik/whoami:latest              52 seconds ago  Up 52 seconds  80/tcp      test-hello
$ ss -tulpn | grep 8080
tcp   LISTEN 0      4096                      *:8080             *:*    users:(("aardvark-dns",pid=47636,fd=3))

Definitions

# test-http.socket
[Unit]
Description=Test HTTP Socket
PartOf=test-main.service

[Socket]
ListenStream=8080
FileDescriptorName=http
Service=test-main.service

[Install]
WantedBy=sockets.target

# test-main.container
[Container]
ContainerName=test-main
Image=docker.io/library/caddy:2.11
Network=test-caddy
Volume=/home/bjorne/.config/test-caddy/Caddyfile:/etc/caddy/Caddyfile:ro
[Unit]
Description=Main container with socket activation
After=network-online.target
Requires=test-http.socket

[Service]
Restart=on-failure
Sockets=test-http.socket

[Install]
WantedBy=default.target

# test-hello.container
[Container]
ContainerName=test-hello
Image=docker.io/traefik/whoami:latest
Network=test-caddy
[Unit]
Description=Second container on same network
After=network-online.target test-main.service

[Service]
Restart=on-failure

[Install]
WantedBy=default.target

# pre-created test-caddy podman network 
➜  ~ podman network inspect test-caddy
[
     {
          "name": "test-caddy",
          "id": "176e9e20b93537664dd1dd03363d6313943081ac4d546df51887077c5df37094",
          "driver": "bridge",
          "network_interface": "podman1",
          "created": "2025-12-31T13:10:28.66241731+01:00",
          "subnets": [
               {
                    "subnet": "10.89.0.0/24",
                    "gateway": "10.89.0.1"
               }
          ],
          "ipv6_enabled": false,
          "internal": false,
          "dns_enabled": true,
          "ipam_options": {
               "driver": "host-local"
          },
          "containers": {
               "c3f2e6915d440a75eb165ec9717cf17646b67d3698a05886ab9f526f2f287341": {
                    "name": "test-hello",
                    "interfaces": {
                         "eth0": {
                              "subnets": [
                                   {
                                        "ipnet": "10.89.0.9/24",
                                        "gateway": "10.89.0.1"
                                   }
                              ],
                              "mac_address": "9e:a1:ff:a9:a9:ef"
                         }
                    }
               }
          }
     }
]

[eriksjolund]

A speculation, maybe this happens:

1. podman inherits the socket-activated socket from systemd.
2. aardvark-dns inherits the socket-activated socket from podman.
3. podman, conmon and the container terminate.
4. systemd tries to bind a new socket to the port but it fails because aardvard-dns is still running so the old socket is still bound to the port.

Maybe step 2 indicates there is a bug in podman?
(Podman could instead start aardvark-dns without passing the socket)

Disclaimer: I did a lot of guessing here. I haven't check this yet.

[bjorne]

Your theory sounds plausible. Any idea how it could be verified?

In the meantime, I think I found an acceptable work around by reducing the coupling between the socket and caddy service, so that a caddy service restart does not lead to a socket restart. This avoids the problem since systemd keeps the socket bound.

My actual setup is a bit different to the examples in this repo and I don't know exactly how this matters for the implied coupling between caddy.service and caddy.socket. (In my setup 80/tcp, 443/tcp and 443/udp are separate .socket files, with distinct FileDescriptorName values http, https and https-quic, in order to utilize named socket file descriptors supported in Caddy 2.11)

Here is how I realized being able to restart caddy.service without restarting caddy-*.socket:

• In container definition, remove Requires= on the socket.
• In socket definitions, remove PartOf= the service.

I will seldom have reason to restart the socket, so I anticipate this avoids the problem nicely. (After reviewing your examples again, I should also make use of caddy reloads instead of caddy restarts!)

Anyway, thanks for providing these examples, they are really great!

[eriksjolund]

Your theory sounds plausible. Any idea how it could be verified?

I opened an issue, see the reproducer

• Restarting socket unit fails with Address already in use because aardvark-dns inherited the socket containers/podman#27854

Regarding how to couple caddy.service and caddy.socket, I'm still a bit exploring that.

[marshallford]

Bug aside, I've been meaning to submit a PR here (or at least a draft) that takes a swing at coupling. My current configuration:

# caddy.socket
[Unit]
PartOf=%N.service

# caddy.container
[Unit]
Requires=%N.socket

This should allow for the service to be stopped/started/enabled without having to deal with the socket directly. @eriksjolund What do you recommend for podman containers + socket activation? Any findings?