Rootless Podman IP address space overlap with host

[jennydaman]

Hi,

I have a project using Podman which is working on Arch Linux (my computer) and Ubuntu (in Github Actions). To further test its portability. I am trying to run Podman in a Fedora Silverblue 37 VM.

After a fresh install of Fedora Silverblue 37 I upgraded Podman to version 4.4.4 by running rpm-ostree upgrade and then systemctl reboot. I made no other changes, I am trying to keep the system as-close to its vendor state as possible. However, I am having an issue with Podman where rootless containers are unable to route to the host.

$ podman run --rm --entrypoint sh \
  docker.io/wbitt/network-multitool@sha256:cbe92911036567a0902838349a5eb2f0147d5341f65e280189e9792323cdbb28 \
  -xc '
    cat /etc/hosts
    traceroute host.containers.internal
    curl host.containers.internal
    ping -c 3 host.containers.internal'
+ cat /etc/hosts
127.0.0.1	localhost localhost.localdomain localhost4 localhost4.localdomain4
::1	localhost localhost.localdomain localhost6 localhost6.localdomain6
10.0.2.15	host.containers.internal
10.0.2.100	78d5b32c0929 awesome_khorana
+ traceroute host.containers.internal
traceroute to host.containers.internal (10.0.2.15), 30 hops max, 60 byte packets
 1  78d5b32c0929 (10.0.2.100)  3049.342 ms !H  3049.330 ms !H  3049.332 ms !H
+ curl host.containers.internal
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:--  0:00:03 --:--:--     0
curl: (7) Failed to connect to host.containers.internal port 80 after 3097 ms: No route to host
+ ping -c 3 host.containers.internal
PING host.containers.internal (10.0.2.15) 56(84) bytes of data.
From 78d5b32c0929 (10.0.2.100) icmp_seq=1 Destination Host Unreachable
From 78d5b32c0929 (10.0.2.100) icmp_seq=2 Destination Host Unreachable
From 78d5b32c0929 (10.0.2.100) icmp_seq=3 Destination Host Unreachable

--- host.containers.internal ping statistics ---
3 packets transmitted, 0 received, +3 errors, 100% packet loss, time 2009ms
pipe 3

It does work with rootful Podman:

$ sudo podman run --rm --entrypoint sh \
  docker.io/wbitt/network-multitool@sha256:cbe92911036567a0902838349a5eb2f0147d5341f65e280189e9792323cdbb28 \
  -xc '
    cat /etc/hosts
    traceroute host.containers.internal
    curl host.containers.internal
    ping -c 3 host.containers.internal'
+ cat /etc/hosts
127.0.0.1	localhost localhost.localdomain localhost4 localhost4.localdomain4
::1	localhost localhost.localdomain localhost6 localhost6.localdomain6
10.88.0.1	host.containers.internal
10.88.0.5	524215401f51 gifted_wilbur
+ traceroute host.containers.internal
traceroute to host.containers.internal (10.88.0.1), 30 hops max, 60 byte packets
 1  host.containers.internal (10.88.0.1)  0.204 ms  0.124 ms  0.115 ms
+ curl host.containers.internal
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
curl: (7) Failed to connect to host.containers.internal port 80 after 0 ms: Connection refused
+ ping -c 3 host.containers.internal
PING host.containers.internal (10.88.0.1) 56(84) bytes of data.
64 bytes from host.containers.internal (10.88.0.1): icmp_seq=1 ttl=64 time=0.024 ms
64 bytes from host.containers.internal (10.88.0.1): icmp_seq=2 ttl=64 time=0.065 ms
64 bytes from host.containers.internal (10.88.0.1): icmp_seq=3 ttl=64 time=0.101 ms

--- host.containers.internal ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 1998ms
rtt min/avg/max/mdev = 0.024/0.063/0.101/0.031 ms

I noticed that the IP address for host.containers.internal is different in rootful mode, so I tried using its IP address as indicated when in rootless mode (same as host's real IP address). It works:

$ sudo podman run --rm --entrypoint sh docker.io/wbitt/network-multitool@sha256:cbe92911036567a0902838349a5eb2f0147d5341f65e280189e9792323cdbb28 -c 'traceroute 10.0.2.15; ping -c 3 10.0.2.15'
traceroute to 10.0.2.15 (10.0.2.15), 30 hops max, 60 byte packets
 1  10.0.2.15 (10.0.2.15)  0.171 ms  0.111 ms  0.104 ms
PING 10.0.2.15 (10.0.2.15) 56(84) bytes of data.
64 bytes from 10.0.2.15: icmp_seq=1 ttl=64 time=0.023 ms
64 bytes from 10.0.2.15: icmp_seq=2 ttl=64 time=0.064 ms
64 bytes from 10.0.2.15: icmp_seq=3 ttl=64 time=0.190 ms

--- 10.0.2.15 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2050ms
rtt min/avg/max/mdev = 0.023/0.092/0.190/0.071 ms

I've tried some various things, none of which worked:

• podman network create other-net --subnet 192.5.0.0/24 and using it
• systemctl stop firewalld
• setenforce 0

Comparing podman info between this Fedora VM and my Arch Linux system, there are no obvious discrepancies besides off-by-one patch version difference for libseccomp

host:
  arch: amd64
  buildahVersion: 1.29.0
  cgroupControllers:
  - cpu
  - io
  - memory
  - pids
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon-2.1.7-2.fc37.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.7, commit: '
  cpuUtilization:
    idlePercent: 94.62
    systemPercent: 1.53
    userPercent: 3.85
  cpus: 20
  distribution:
    distribution: fedora
    variant: silverblue
    version: "37"
  eventLogger: journald
  hostname: fedora
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
  kernel: 6.2.9-200.fc37.x86_64
  linkmode: dynamic
  logDriver: journald
  memFree: 2105700352
  memTotal: 8308289536
  networkBackend: netavark
  ociRuntime:
    name: crun
    package: crun-1.8.3-2.fc37.x86_64
    path: /usr/bin/crun
    version: |-
      crun version 1.8.3
      commit: 59f2beb7efb0d35611d5818fd0311883676f6f7e
      rundir: /run/user/1000/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +LIBKRUN +WASM:wasmedge +YAJL
  os: linux
  remoteSocket:
    exists: true
    path: /run/user/1000/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: true
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.2.0-8.fc37.x86_64
    version: |-
      slirp4netns version 1.2.0
      commit: 656041d45cfca7a4176f6b7eed9e4fe6c11e8383
      libslirp: 4.7.0
      SLIRP_CONFIG_VERSION_MAX: 4
      libseccomp: 2.5.3
  swapFree: 6464729088
  swapTotal: 8307863552
  uptime: 1h 60m 53.00s (Approximately 0.04 days)
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  volume:
  - local
registries:
  search:
  - registry.fedoraproject.org
  - registry.access.redhat.com
  - docker.io
  - quay.io
store:
  configFile: /var/home/chris/.config/containers/storage.conf
  containerStore:
    number: 18
    paused: 0
    running: 14
    stopped: 4
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /var/home/chris/.local/share/containers/storage
  graphRootAllocated: 41872785408
  graphRootUsed: 17441189888
  graphStatus:
    Backing Filesystem: btrfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 13
  runRoot: /run/user/1000/containers
  transientStore: false
  volumePath: /var/home/chris/.local/share/containers/storage/volumes
version:
  APIVersion: 4.4.4
  Built: 1680521485
  BuiltTime: Mon Apr  3 07:31:25 2023
  GitCommit: ""
  GoVersion: go1.19.7
  Os: linux
  OsArch: linux/amd64
  Version: 4.4.4

How else can I debug?

[Luap99]

I assume your host ip is 10.0.2.15.
You cannot reach ips in the 10.0.2.0/24 by default because this is the default subnet which is used by slirp4netns in the container.
You have to change the slirp4netns subnet to something different, see https://blog.podman.io/2023/02/default-podman-network-subnets/

[jennydaman]

Thank you for the fast reply! That was indeed the problem, though instead of changing Podman's configuration, I tried recreating my Silverblue VM with a different IP address.

I wonder why it worked in rootfull mode, is that a different capability of netavark? Oh well...