Rootless quadlet network 'unstable'?

[haleksandre]

I've been able to boot an FCOS with rootless quadlet-generated containers. The problem seems to be that the containers within the network (bridge) are having intermittent connectivity which makes the containers more often than not unable to connect to each other.

My Butane config looks something like so

variant: fcos
version: 1.5.0
passwd:
  users:
    - name: core
      groups:
        - sudo
        - docker
      # password_hash: # ....hash
      # ssh_authorized_keys:
        # ...keys
      home_dir: /home/core
storage:
  #  disks: ...
  # filesystems: ...
  # directories: ...
  files:
    # ...files
    - path: /etc/containers/systemd/users/database.network
      contents:
        inline: |
          [Network]
          Label=lan.media.Network=database.network
    - path: /etc/containers/systemd/users/server.network
      contents:
        inline: |
          [Network]
          Label=lan.media.Network=server.network
    - path: /etc/containers/systemd/users/nginx.volume
      contents:
        inline: |
          [Volume]
          Label=lan.media.volume=nginx
    - path: /etc/containers/systemd/users/postgres.container
      contents:
        inline: |
          [Unit]
          Description=Postgres Quadlet

          [Container]
          Image=docker.io/library/postgres:15.4
          ContainerName=postgres
          AutoUpdate=registry
          User=root
          Network=database.network
          Environment=POSTGRES_PASSWORD=${POSTGRES_PASSWORD}
          Volume=postgres.volume:/var/lib/postgresql/data:z
          Volume=/etc/postgres/conf.d:/etc/postgres/conf.d:z
          Volume=/var/srv/postgres/15.4/docker-entrypoint-initdb.d:/docker-entrypoint-initdb.d:z

          [Service]
          Restart=always
          TimeoutStartSec=900

          [Install]
          WantedBy=multi-user.target default.target
    - path: /etc/containers/systemd/users/dragonfly.container
      contents:
        inline: |
          [Unit]
          Description=Dragonfly Quadlet

          [Container]
          Image=docker.dragonflydb.io/dragonflydb/dragonfly:latest
          ContainerName=dragonfly
          AutoUpdate=registry
          User=root
          Network=database.network
          PublishPort=6379:6379/tcp
          Volume=dragonfly.volume:/data:z
          SeccompProfile=unconfined

          [Service]
          Restart=always
          TimeoutStartSec=900

          [Install]
          WantedBy=multi-user.target default.target
    - path: /etc/containers/systemd/users/nginx.container
      contents:
        inline: |
          [Unit]
          Description=Nginx Quadlet
          Requires=podman.socket
          After=podman.socket

          [Container]
          Image=docker.io/nginxproxy/nginx-proxy:alpine
          ContainerName=nginx
          AutoUpdate=registry
          Environment=DOCKER_HOST=unix://${XDG_RUNTIME_DIR}/podman/podman.sock
          Network=server.network
          PublishPort=80:80/tcp
          PublishPort=443:443/tcp
          Volume=${XDG_RUNTIME_DIR}/podman/podman.sock:/tmp/docker.sock:ro
          Volume=nginx.volume:/usr/share/nginx/html
          Volume=/etc/nginx/conf.d/:/etc/nginx/conf.d/
          Volume=/etc/nginx/global/:/etc/nginx/global/
          Volume=/etc/nginx/ssl/:/etc/nginx/ssl/

          [Service]
          Restart=always
          TimeoutStartSec=900

          [Install]
          WantedBy=multi-user.target
    - path: /etc/containers/systemd/users/immich.container
      contents:
        inline: |
          [Unit]
          Description=Immich Quadlet

          [Container]
          Image=ghcr.io/imagegenius/immich:latest
          ContainerName=immich
          AutoUpdate=registry
          Network=database.network
          Network=server.network
          EnvironmentFile=/var/srv/immich/latest/.env
          Environment=VIRTUAL_HOST=host
          Environment=VIRTUAL_PORT=8080
          PublishPort=8090:8080/tcp
          Volume=/var/srv/immich/config:/config:z
          Volume=/var/srv/immich/machine-learning:/config/machine-learning:z
          Volume=/var/media/photos:/photos:z

          [Service]
          Restart=always
          TimeoutStartSec=900

          [Install]
          WantedBy=multi-user.target default.target
      # ...other quadlet
# systems:
  #  units:

In this simplified example above, Immich is having a hard time connecting to Postgres and Dragonfly which is making it difficult to use as you have to refresh or send a request that saves something to the db numerous times so that it will eventually connect & succeed.

The rootless containers are using a non-root inside, could it be the reason for preventing the containers from occasionally connecting to each other? If so, how can I make it more 'stable'? It's confusing that it's working sometimes & not other times..

[ygalblum]

I'm trying to understand where the issue starts - Quadlet yes or no, rootless yes or no, multiple networks of not. So I want to look into the following:

1. Did you try running the same without Quadlet - that is run this deployment manually?
2. Can you please elaborate on what you mean by "sometimes it works but sometime it doesn't"? Is it separate boots? Is it separate cycles of service restart? Sporadic?
3. Is the issue specific to the container attached to both networks? Do you see other communication issues?
4. Can you please try the same containers but connect all of them to the a single network?

[haleksandre]

I have not tried without Quadlet yet, as I am provisioning an FCOS machine with the ignition config above. The configuration creates a few rootless containers, networks & volumes. The rootless containers themselves run via a rootless user within, which I know makes it so the visible UID is another non-root UID on the host & not my own UID 1000. Could this be an issue with permissions or something?

As for the connectivity issues, the containers have sporadic connectivity between themselves & from the look of it even the outside network (internet). The rootless containers can't seem to find the other container's internal IP via their alias.

An example of the kind of errors the rootless container gets while trying to connect to the other container named dragonfly:

Error: getaddrinfo ENOTFOUND dragonfly
    at GetAddrInfoReqWrap.onlookup [as oncomplete] (node:dns:108:26) {
  errno: -3008,
  code: 'ENOTFOUND',
  syscall: 'getaddrinfo',
  hostname: 'dragonfly'
}

After adding the flag --user-ns keep-id, the rootless containers running as a UID 1000 seem better. The connectivity issues seem to happen less frequently, but it is still present. Could it have something to do with slirp4netns network & permissions?

[haleksandre]

I was able to run some tests recently & still haven't found exactly what the issue is. In the containers, if I ping an address I, which seems randomly, either receive a bad address error or it pings successfully.

ping: bad address 'google.com'
7ad3338c4b2c:/# ping google.com
PING google.com (142.251.41.78): 56 data bytes
64 bytes from 142.251.41.78: seq=0 ttl=42 time=25.240 ms
64 bytes from 142.251.41.78: seq=1 ttl=42 time=25.218 ms
64 bytes from 142.251.41.78: seq=2 ttl=42 time=24.658 ms
64 bytes from 142.251.41.78: seq=3 ttl=42 time=30.013 ms
^C
--- google.com ping statistics ---
4 packets transmitted, 4 packets received, 0% packet loss
round-trip min/avg/max = 24.658/26.282/30.013 ms
7ad3338c4b2c:/# ping google.com
ping: bad address 'google.com'
7ad3338c4b2c:/# ping google.com
PING google.com (142.251.41.46): 56 data bytes
64 bytes from 142.251.41.46: seq=0 ttl=42 time=26.792 ms
64 bytes from 142.251.41.46: seq=1 ttl=42 time=26.764 ms
64 bytes from 142.251.41.46: seq=2 ttl=42 time=26.373 ms
64 bytes from 142.251.41.46: seq=3 ttl=42 time=26.143 ms
^C
--- google.com ping statistics ---
4 packets transmitted, 4 packets received, 0% packet loss
round-trip min/avg/max = 26.143/26.518/26.792 ms
7ad3338c4b2c:/# ping google.com
PING google.com (142.251.41.46): 56 data bytes
64 bytes from 142.251.41.46: seq=0 ttl=42 time=27.463 ms
64 bytes from 142.251.41.46: seq=1 ttl=42 time=26.953 ms
64 bytes from 142.251.41.46: seq=2 ttl=42 time=26.990 ms
64 bytes from 142.251.41.46: seq=3 ttl=42 time=26.798 ms
^C
--- google.com ping statistics ---
4 packets transmitted, 4 packets received, 0% packet loss
round-trip min/avg/max = 26.798/27.051/27.463 ms
7ad3338c4b2c:/# ping google.com
ping: bad address 'google.com'

It looks like the network within the containers errors & then restarts & works for a couple of seconds & then errors & the process restarts. Do you have any ideas on how I can troubleshoot this further or why & what could be doing this?

[rhatdan]

@Luap99 any ideas?

[Luap99]

Are you using cni or netavark (check podman info --format {{.Host.NetworkBackend}} )?

If it is netavark then you likely hit a issue similar to containers/aardvark-dns#389 if you have any starting/stopping containers while this happens.

[haleksandre]

I use netavark, but no container is starting or stopping. They are all either started (healthy) or stopped. Even when I stop & remove all the other containers that aren't needed to run this particular container, to make sure, I still encounter the random bad request error.