Rootless podman in rootless podman running server permission denied

[DomenPigeon]

Issue Description

Rootless podman in rootless podman doesn't allow ports when running a container.

Steps to reproduce the issue

podman run --security-opt label=disable --user podman -it --device /dev/fuse quay.io/podman/stable podman run docker.io/library/nginx

Describe the results you received

bind() to [0.0.0.0:80](http://0.0.0.0/) failed (13: Permission denied)
nginx: [emerg] bind() to [0.0.0.0:80](http://0.0.0.0/) failed (13: Permission denied)

Describe the results you expected

Nginx would start normally and run.

podman info output

Podman in podman info:
[podman@e0a8e0e6a81a /]$ podman info
host:
  arch: amd64
  buildahVersion: 1.35.3
  cgroupControllers:
  - cpu
  - io
  - memory
  - pids
  cgroupManager: cgroupfs
  cgroupVersion: v2
  conmon:
    package: conmon-2.1.10-1.fc40.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.10, commit: '
  cpuUtilization:
    idlePercent: 99.05
    systemPercent: 0.39
    userPercent: 0.56
  cpus: 32
  databaseBackend: sqlite
  distribution:
    distribution: fedora
    variant: container
    version: "40"
  eventLogger: file
  freeLocks: 2047
  hostname: e0a8e0e6a81a
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 1
      size: 999
    - container_id: 1000
      host_id: 1001
      size: 64535
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 1
      size: 999
    - container_id: 1000
      host_id: 1001
      size: 64535
  kernel: 6.8.5-301.fc40.x86_64
  linkmode: dynamic
  logDriver: k8s-file
  memFree: 57368600576
  memTotal: 66503323648
  networkBackend: netavark
  networkBackendInfo:
    backend: netavark
    dns:
      package: aardvark-dns-1.10.0-1.fc40.x86_64
      path: /usr/libexec/podman/aardvark-dns
      version: aardvark-dns 1.10.0
    package: netavark-1.10.3-3.fc40.x86_64
    path: /usr/libexec/podman/netavark
    version: netavark 1.10.3
  ociRuntime:
    name: crun
    package: crun-1.14.4-1.fc40.x86_64
    path: /usr/bin/crun
    version: |-
      crun version 1.14.4
      commit: a220ca661ce078f2c37b38c92e66cf66c012d9c1
      rundir: /tmp/storage-run-1000/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +LIBKRUN +WASM:wasmedge +YAJL
  os: linux
  pasta:
    executable: /usr/bin/pasta
    package: passt-0^20240426.gd03c4e2-1.fc40.x86_64
    version: |
      pasta 0^20240426.gd03c4e2-1.fc40.x86_64
      Copyright Red Hat
      GNU General Public License, version 2 or later
        <https://www.gnu.org/licenses/old-licenses/gpl-2.0.html>
      This is free software: you are free to change and redistribute it.
      There is NO WARRANTY, to the extent permitted by law.
  remoteSocket:
    exists: false
    path: /tmp/storage-run-1000/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: false
  serviceIsRemote: false
  slirp4netns:
    executable: ""
    package: ""
    version: ""
  swapFree: 8589930496
  swapTotal: 8589930496
  uptime: 0h 8m 22.00s
  variant: ""
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries:
  search:
  - registry.fedoraproject.org
  - registry.access.redhat.com
  - docker.io
  - quay.io
store:
  configFile: /home/podman/.config/containers/storage.conf
  containerStore:
    number: 1
    paused: 0
    running: 0
    stopped: 1
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /home/podman/.local/share/containers/storage
  graphRootAllocated: 8350035968
  graphRootUsed: 7404474368
  graphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Supports shifting: "false"
    Supports volatile: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 1
  runRoot: /tmp/storage-run-1000/containers
  transientStore: false
  volumePath: /home/podman/.local/share/containers/storage/volumes
version:
  APIVersion: 5.0.2
  Built: 1713312000
  BuiltTime: Wed Apr 17 00:00:00 2024
  GitCommit: ""
  GoVersion: go1.22.1
  Os: linux
  OsArch: linux/amd64
  Version: 5.0.2



Podman info:
liveuser@localhost-live:~$ podman version
Client:       Podman Engine
Version:      5.0.3
API Version:  5.0.3
Go Version:   go1.22.2
Built:        Thu May  9 20:00:00 2024
OS/Arch:      linux/amd64

liveuser@localhost-live:~$ podman info
host:
  arch: amd64
  buildahVersion: 1.35.4
  cgroupControllers:
  - cpu
  - io
  - memory
  - pids
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon-2.1.10-1.fc40.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.10, commit: '
  cpuUtilization:
    idlePercent: 99.09
    systemPercent: 0.37
    userPercent: 0.54
  cpus: 32
  databaseBackend: sqlite
  distribution:
    distribution: fedora
    variant: workstation
    version: "40"
  eventLogger: journald
  freeLocks: 2042
  hostname: localhost-live
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 524288
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 524288
      size: 65536
  kernel: 6.8.5-301.fc40.x86_64
  linkmode: dynamic
  logDriver: journald
  memFree: 57458049024
  memTotal: 66503323648
  networkBackend: netavark
  networkBackendInfo:
    backend: netavark
    dns:
      package: aardvark-dns-1.10.0-1.fc40.x86_64
      path: /usr/libexec/podman/aardvark-dns
      version: aardvark-dns 1.10.0
    package: netavark-1.10.3-3.fc40.x86_64
    path: /usr/libexec/podman/netavark
    version: netavark 1.10.3
  ociRuntime:
    name: crun
    package: crun-1.14.4-1.fc40.x86_64
    path: /usr/bin/crun
    version: |-
      crun version 1.14.4
      commit: a220ca661ce078f2c37b38c92e66cf66c012d9c1
      rundir: /run/user/1000/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +LIBKRUN +WASM:wasmedge +YAJL
  os: linux
  pasta:
    executable: /usr/bin/pasta
    package: passt-0^20240326.g4988e2b-1.fc40.x86_64
    version: |
      pasta 0^20240326.g4988e2b-1.fc40.x86_64
      Copyright Red Hat
      GNU General Public License, version 2 or later
        <https://www.gnu.org/licenses/old-licenses/gpl-2.0.html>
      This is free software: you are free to change and redistribute it.
      There is NO WARRANTY, to the extent permitted by law.
  remoteSocket:
    exists: false
    path: /run/user/1000/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: true
  serviceIsRemote: false
  slirp4netns:
    executable: ""
    package: ""
    version: ""
  swapFree: 8589930496
  swapTotal: 8589930496
  uptime: 0h 9m 12.00s
  variant: ""
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries:
  search:
  - registry.fedoraproject.org
  - registry.access.redhat.com
  - docker.io
  - quay.io
store:
  configFile: /home/liveuser/.config/containers/storage.conf
  containerStore:
    number: 2
    paused: 0
    running: 0
    stopped: 2
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /home/liveuser/.local/share/containers/storage
  graphRootAllocated: 8350035968
  graphRootUsed: 7404638208
  graphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Supports shifting: "false"
    Supports volatile: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 1
  runRoot: /run/user/1000/containers
  transientStore: false
  volumePath: /home/liveuser/.local/share/containers/storage/volumes
version:
  APIVersion: 5.0.3
  Built: 1715299200
  BuiltTime: Thu May  9 20:00:00 2024
  GitCommit: ""
  GoVersion: go1.22.2
  Os: linux
  OsArch: linux/amd64
  Version: 5.0.3

Podman in a container

Yes

Privileged Or Rootless

Rootless

Upstream Latest Release

No

Additional environment details

I have tried this on WSL2, Ubuntu with different versions of podman but the result was always the same. For this issue I have also tried on fedora with latest podman to see if it was and OS problem, but the result was the same.
Tried also witha appache httpd but the result was the same.

Additional information

Additional information like issue happens only occasionally or issue happens with a particular architecture or on a particular setting

[Luap99]

The podman image uses --network=host as default for its containers and because you run as normal users it cannot bind a privileged port. So either you set --network bridge and forward a port or you can set --sysctl net.ipv4.ip_unprivileged_port_start=80 on the outer container.

In any case this is not bug so I move it to a discussion.

[DomenPigeon]

Ah ofcourse, thanks for the explanation @Luap99

[DomenPigeon]

@Luap99 I still cannot get it to work, with exposed port. If I try docker.io/apache/apisix image for example which uses port 9080 it works and the container is running, but the ports are not exposed and accessible.

If I run:
podman run --security-opt label=disable --user podman -it --device /dev/fuse quay.io/podman/stable
And log into the container:
podman run -p 6000:9080 -d docker.io/apache/apisix
The apisix is running but the port isn't exposed, curl localhost:6000 inside the container gives an error.

If there are any resources for this you can just point me there.

[DomenPigeon]

@Luap99 On second thought, I still don't understand why are the server permission denied, beucase mapping to an unprivileged port also doesn't work
podman run --security-opt label=disable --user podman -it --device /dev/fuse quay.io/podman/stable podman run -p 5000:80 docker.io/library/nginx
This gives the same result.

[Luap99]

as written above you have to use --network bridge on the inner container as defaults to host otherwise