rootless podman on Fedora - "potentially insufficient UIDs or GIDs available in user namespace"

[bfournie]

Issue Description

I know this issue has been seen before, but none of the solutions seem to fit the problem I am having.
I am running on Fedora 42

I am trying to do a podman build using the command:
$ podman build --authfile ./pull-secret.json --secret id=my_pull_secret,src=./pull-secret.json -t my-iso-builder:latest -f Dockerfile

The Dockerfile executes a script which does a podman run in order to create an ISO. This is the command in the script:
$ podman run --authfile "${PULL_SECRET_FILE}" --rm -it --privileged --pull newer --net=host -v "${appliance_work_dir}"/:/assets:Z "${appliance_image}" build live-iso --log-level debug

When the podman in the script runs I get the error:

Building appliance ISO (image: registry.ci.openshift.org/ocp/4.20:agent-preinstall-image-builder)
time="2025-08-04T18:50:29Z" level=warning msg="Using rootless single mapping into the namespace. This might break some images. Check /etc/subuid and /etc/subgid for adding sub*ids if not using a network user"
time="2025-08-04T18:50:29Z" level=warning msg="The input device is not a TTY. The --tty and --interactive flags might not work properly"
Trying to pull registry.ci.openshift.org/ocp/4.20:agent-preinstall-image-builder...
Getting image source signatures
Copying blob sha256:db5ac9e2b94214a23a3e7dd6a518358bc118d5304cbfba7e0250b7af1f3daf1d
Copying blob sha256:b5ddbc70c93efc07a634d09a4c598e797fca8fc7b9a13c225b3055b380b9fa70
Error: copying system image from manifest list: writing blob: adding layer with blob "sha256:b5ddbc70c93efc07a634d09a4c598e797fca8fc7b9a13c225b3055b380b9fa70"/""/"sha256:c79f3694349ffe3805aeff70f88a967e953efd3664e6804bbca71b72cdf3f5f1": unpacking failed (error: exit status 1; output: potentially insufficient UIDs or GIDs available in user namespace (requested 0:12 for /var/spool/mail): Check /etc/subuid and /etc/subgid if configured locally and run "podman system migrate": lchown /var/spool/mail: invalid argument)
Error: building at STEP "RUN --mount=type=secret,id=my_pull_secret,target=${PULL_SECRET_CONTAINER_PATH} hack/build-ove-image.sh --pull-secret-file "${PULL_SECRET_CONTAINER_PATH}" --ocp-version 4.20.0": while running runtime: exit status 125

My uids should be adequate, but its very possible there is an issue here:

$ grep "$(whoami)" /etc/subgid
bfournie:100000:262144
$ grep "$(whoami)" /etc/subuid
bfournie:100000:262144

Steps to reproduce the issue

Steps to reproduce the issue

1. Run the podman build command as above

Describe the results you received

Podman build fails with uid error as above

Describe the results you expected

Podman build succeeds.

podman info output

$ podman info
host:
  arch: amd64
  buildahVersion: 1.40.1
  cgroupControllers:
  - cpu
  - io
  - memory
  - pids
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon-2.1.13-1.fc42.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.13, commit: '
  cpuUtilization:
    idlePercent: 94.52
    systemPercent: 1.66
    userPercent: 3.83
  cpus: 8
  databaseBackend: sqlite
  distribution:
    distribution: fedora
    variant: workstation
    version: "42"
  eventLogger: journald
  freeLocks: 2044
  hostname: bfournie-thinkpadx1carbongen9.westford.csb
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 29373
      size: 1
    - container_id: 1
      host_id: 100000
      size: 262144
    uidmap:
    - container_id: 0
      host_id: 29373
      size: 1
    - container_id: 1
      host_id: 100000
      size: 262144
  kernel: 6.15.8-200.fc42.x86_64
  linkmode: dynamic
  logDriver: journald
  memFree: 19247517696
  memTotal: 33338863616
  networkBackend: netavark
  networkBackendInfo:
    backend: netavark
    dns:
      package: aardvark-dns-1.15.0-1.fc42.x86_64
      path: /usr/libexec/podman/aardvark-dns
      version: aardvark-dns 1.15.0
    package: netavark-1.15.2-1.fc42.x86_64
    path: /usr/libexec/podman/netavark
    version: netavark 1.15.2
  ociRuntime:
    name: crun
    package: crun-1.23.1-1.fc42.x86_64
    path: /usr/bin/crun
    version: |-
      crun version 1.23.1
      commit: d20b23dba05e822b93b82f2f34fd5dada433e0c2
      rundir: /run/user/29373/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +LIBKRUN +WASM:wasmedge +YAJL
  os: linux
  pasta:
    executable: /usr/bin/pasta
    package: passt-0^20250611.g0293c6f-1.fc42.x86_64
    version: |
      pasta 0^20250611.g0293c6f-1.fc42.x86_64
      Copyright Red Hat
      GNU General Public License, version 2 or later
        <https://www.gnu.org/licenses/old-licenses/gpl-2.0.html>
      This is free software: you are free to change and redistribute it.
      There is NO WARRANTY, to the extent permitted by law.
  remoteSocket:
    exists: true
    path: /run/user/29373/podman/podman.sock
  rootlessNetworkCmd: pasta
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: true
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.3.1-2.fc42.x86_64
    version: |-
      slirp4netns version 1.3.1
      commit: e5e368c4f5db6ae75c2fce786e31eef9da6bf236
      libslirp: 4.8.0
      SLIRP_CONFIG_VERSION_MAX: 5
      libseccomp: 2.5.5
  swapFree: 8589930496
  swapTotal: 8589930496
  uptime: 2h 25m 24.00s (Approximately 0.08 days)
  variant: ""
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries:
  search:
  - registry.fedoraproject.org
  - registry.access.redhat.com
  - docker.io
store:
  configFile: /home/bfournie/.config/containers/storage.conf
  containerStore:
    number: 0
    paused: 0
    running: 0
    stopped: 0
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /home/bfournie/.local/share/containers/storage
  graphRootAllocated: 510389125120
  graphRootUsed: 68887924736
  graphStatus:
    Backing Filesystem: btrfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Supports shifting: "false"
    Supports volatile: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 10
  runRoot: /run/user/29373/containers
  transientStore: false
  volumePath: /home/bfournie/.local/share/containers/storage/volumes
version:
  APIVersion: 5.5.2
  BuildOrigin: Fedora Project
  Built: 1750723200
  BuiltTime: Mon Jun 23 20:00:00 2025
  GitCommit: e7d8226745ba07a64b7176a7f128e4ef53225a0e
  GoVersion: go1.24.4
  Os: linux
  OsArch: linux/amd64
  Version: 5.5.2

Podman in a container

Yes

Privileged Or Rootless

Rootless

Upstream Latest Release

Yes

Additional environment details

$ cat /etc/redhat-release
Fedora release 42 (Adams)

Additional information

I don't get the issue if I run sudo podman build, however this results in various other problems that I started to debug but thought I would focus on this uid issue first.

Thank you for any help.

[bfournie]

Some more info with debug:

$ podman info --debug
host:
  arch: amd64
  buildahVersion: 1.40.1
  cgroupControllers:
  - cpu
  - io
  - memory
  - pids
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon-2.1.13-1.fc42.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.13, commit: '
  cpuUtilization:
    idlePercent: 95.96
    systemPercent: 1.18
    userPercent: 2.85
  cpus: 8
  databaseBackend: sqlite
  distribution:
    distribution: fedora
    variant: workstation
    version: "42"
  eventLogger: journald
  freeLocks: 2044
  hostname: bfournie-thinkpadx1carbongen9.westford.csb
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 29373
      size: 1
    - container_id: 1
      host_id: 100000
      size: 262144
    uidmap:
    - container_id: 0
      host_id: 29373
      size: 1
    - container_id: 1
      host_id: 100000
      size: 262144
  kernel: 6.15.8-200.fc42.x86_64
  linkmode: dynamic
  logDriver: journald
  memFree: 18995097600
  memTotal: 33338863616
  networkBackend: netavark
  networkBackendInfo:
    backend: netavark
    dns:
      package: aardvark-dns-1.15.0-1.fc42.x86_64
      path: /usr/libexec/podman/aardvark-dns
      version: aardvark-dns 1.15.0
    package: netavark-1.15.2-1.fc42.x86_64
    path: /usr/libexec/podman/netavark
    version: netavark 1.15.2
  ociRuntime:
    name: crun
    package: crun-1.23.1-1.fc42.x86_64
    path: /usr/bin/crun
    version: |-
      crun version 1.23.1
      commit: d20b23dba05e822b93b82f2f34fd5dada433e0c2
      rundir: /run/user/29373/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +LIBKRUN +WASM:wasmedge +YAJL
  os: linux
  pasta:
    executable: /usr/bin/pasta
    package: passt-0^20250611.g0293c6f-1.fc42.x86_64
    version: |
      pasta 0^20250611.g0293c6f-1.fc42.x86_64
      Copyright Red Hat
      GNU General Public License, version 2 or later
        <https://www.gnu.org/licenses/old-licenses/gpl-2.0.html>
      This is free software: you are free to change and redistribute it.
      There is NO WARRANTY, to the extent permitted by law.
  remoteSocket:
    exists: true
    path: /run/user/29373/podman/podman.sock
  rootlessNetworkCmd: pasta
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: true
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.3.1-2.fc42.x86_64
    version: |-
      slirp4netns version 1.3.1
      commit: e5e368c4f5db6ae75c2fce786e31eef9da6bf236
      libslirp: 4.8.0
      SLIRP_CONFIG_VERSION_MAX: 5
      libseccomp: 2.5.5
  swapFree: 8589930496
  swapTotal: 8589930496
  uptime: 4h 1m 31.00s (Approximately 0.17 days)
  variant: ""
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries:
  search:
  - registry.fedoraproject.org
  - registry.access.redhat.com
  - docker.io
store:
  configFile: /home/bfournie/.config/containers/storage.conf
  containerStore:
    number: 0
    paused: 0
    running: 0
    stopped: 0
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /home/bfournie/.local/share/containers/storage
  graphRootAllocated: 510389125120
  graphRootUsed: 68880977920
  graphStatus:
    Backing Filesystem: btrfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Supports shifting: "false"
    Supports volatile: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 10
  runRoot: /run/user/29373/containers
  transientStore: false
  volumePath: /home/bfournie/.local/share/containers/storage/volumes
version:
  APIVersion: 5.5.2
  BuildOrigin: Fedora Project
  Built: 1750723200
  BuiltTime: Mon Jun 23 20:00:00 2025
  GitCommit: e7d8226745ba07a64b7176a7f128e4ef53225a0e
  GoVersion: go1.24.4
  Os: linux
  OsArch: linux/amd64
  Version: 5.5.2

$ ls -ld ~/.local/share/containers/storage
drwx------. 1 bfournie bfournie 248 Aug  4 14:49 /home/bfournie/.local/share/containers/storage

I also have a minimal reproducer here using this Dockerfile:

FROM registry.access.redhat.com/ubi9/ubi:9.6-1753978585

RUN dnf install -y podman && dnf clean all

RUN --mount=type=secret,id=my_pull_secret,target=/run/secrets/my_pull_secret_file \
    podman pull --authfile /run/secrets/my_pull_secret_file registry.ci.openshift.org/ocp/4.20:agent-preinstall-image-builder

Running this command I get the failure:

$ podman build --authfile ./pull-secret.json --secret id=my_pull_secret,src=./pull-secret.json -t my-iso-builder:latest -f Dockerfile.simple 
STEP 1/3: FROM registry.access.redhat.com/ubi9/ubi:9.6-1753978585
STEP 2/3: RUN dnf install -y podman && dnf clean all
--> Using cache a52c8f8f47aa23ae7e9ca1c572acfceac2d87bd66ae8053edf32eb0d54144baa
--> a52c8f8f47aa
STEP 3/3: RUN --mount=type=secret,id=my_pull_secret,target=/run/secrets/my_pull_secret_file     podman pull --authfile /run/secrets/my_pull_secret_file registry.ci.openshift.org/ocp/4.20:agent-preinstall-image-builder
time="2025-08-04T20:49:54Z" level=warning msg="Using rootless single mapping into the namespace. This might break some images. Check /etc/subuid and /etc/subgid for adding sub*ids if not using a network user"
Trying to pull registry.ci.openshift.org/ocp/4.20:agent-preinstall-image-builder...
Getting image source signatures
Copying blob sha256:db5ac9e2b94214a23a3e7dd6a518358bc118d5304cbfba7e0250b7af1f3daf1d
Copying blob sha256:b5ddbc70c93efc07a634d09a4c598e797fca8fc7b9a13c225b3055b380b9fa70
Error: copying system image from manifest list: writing blob: adding layer with blob "sha256:b5ddbc70c93efc07a634d09a4c598e797fca8fc7b9a13c225b3055b380b9fa70"/""/"sha256:c79f3694349ffe3805aeff70f88a967e953efd3664e6804bbca71b72cdf3f5f1": unpacking failed (error: exit status 1; output: potentially insufficient UIDs or GIDs available in user namespace (requested 0:12 for /var/spool/mail): Check /etc/subuid and /etc/subgid if configured locally and run "podman system migrate": lchown /var/spool/mail: invalid argument)
Error: building at STEP "RUN --mount=type=secret,id=my_pull_secret,target=/run/secrets/my_pull_secret_file podman pull --authfile /run/secrets/my_pull_secret_file registry.ci.openshift.org/ocp/4.20:agent-preinstall-image-builder": while running runtime: exit status 125

[sbrivio-rh]

Could it be that you're hitting https://bugzilla.redhat.com/show_bug.cgi?id=2382662, that is, that you had https://src.fedoraproject.org/rpms/shadow-utils/c/e1cfa31731cd68aa2d76ebfb3142037d16d3bb1e applied at the time you created the bfournie user?

[Luap99]

Could it be that you're hitting https://bugzilla.redhat.com/show_bug.cgi?id=2382662, that is, that you had https://src.fedoraproject.org/rpms/shadow-utils/c/e1cfa31731cd68aa2d76ebfb3142037d16d3bb1e applied at the time you created the bfournie user?

No that is unrelated most likely.

The issue here is you are trying to run podman withing podman which is complex and requires special setup to get it to work. You have subuid mappings on the host for th eusr which is why you can use it normally and run build. However if you try to use podman within podman the build container would also need mappings defined inside for the user in the build container.
The easiest answer is likely don't do that, run the podman run iso step on the host and then copy the iso into the build.

As for configuring podman to run within a container see https://www.redhat.com/en/blog/podman-inside-container

In any case I don't see a bug here so I move it to a discussion.

[bfournie]

Thanks so much @Luap99 and @sbrivio-rh for your help, and for providing the link to the doc. My goal is to run podman (rootful or rootless) in rootless podman, but if that's not possible I can try running in rootful podman.

Using the example from the doc, I'm still hitting the same issue related to the uuid for /var/spool/mail. (potentially insufficient UIDs or GIDs available in user namespace (requested 0:12 for /var/spool/mail)

Here is my Dockerfile based on the example:

FROM registry.fedoraproject.org/fedora:latest

RUN dnf -y update; yum -y reinstall shadow-utils; \
yum -y install podman fuse-overlayfs --exclude container-selinux; \
rm -rf /var/cache /var/log/dnf* /var/log/yum.*

RUN useradd podman; \
echo podman:100000:65536 > /etc/subuid; \
echo podman:100000:65536 > /etc/subgid;

VOLUME /var/lib/containers
VOLUME /home/podman/.local/share/containers

# COPY the file from the build context into the container
COPY containers.conf /etc/containers/containers.conf
COPY containers.conf /home/podman/.config/containers/containers.conf

COPY storage.conf /etc/containers/storage.conf

RUN chown podman:podman -R /home/podman

# chmod containers.conf and adjust storage.conf to enable Fuse storage.
RUN chmod 644 /etc/containers/containers.conf
RUN mkdir -p /var/lib/shared/overlay-images /var/lib/shared/overlay-layers /var/lib/shared/vfs-images /var/lib/shared/vfs-layers; touch /var/lib/shared/overlay-images/images.lock; touch /var/lib/shared/overlay-layers/layers.lock; touch /var/lib/shared/vfs-images/images.lock; touch /var/lib/shared/vfs-layers/layers.lock

ENV _CONTAINERS_USERNS_CONFIGURED=""

RUN podman run --privileged quay.io/podman/stable podman run ubi8 echo hello

I'm hitting the same error related to the /var/spool/mail uid

STEP 13/13: RUN podman run --privileged quay.io/podman/stable podman run ubi8 echo hello
time="2025-08-05T13:35:35Z" level=warning msg="Using rootless single mapping into the namespace. This might break some images. Check /etc/subuid and /etc/subgid for adding sub*ids if not using a network user"
Trying to pull quay.io/podman/stable:latest...
Getting image source signatures
...
Error: internal error: unable to copy from source docker://quay.io/podman/stable:latest: copying system image from manifest list: writing blob: adding layer with blob "sha256:6dc3a7bd47c5d1ff88b30de4b9b426dcf0bc116bdd3c1666907f0c52fd4c1cec"/""/"sha256:41805d31fc6db139486f254c9d85c2847525469068b9ce64cf8a8bba84d99418": unpacking failed (error: exit status 1; output: potentially insufficient UIDs or GIDs available in user namespace (requested 0:12 for /var/spool/mail): Check /etc/subuid and /etc/subgid if configured locally and run "podman system migrate": lchown /var/spool/mail: invalid argument)
Error: building at STEP "RUN podman run --privileged quay.io/podman/stable podman run ubi8 echo hello": while running runtime: exit status 125

[bfournie]

I guess somehow I need to map this uid/gid on the host but its not clear how to do that
requested 0:12 for /var/spool/mail

[bfournie]

$ cat /etc/subuid
bfournie:100000:262144
$ cat /etc/subgid
$ podman run fedora cat /proc/self/uid_map
         0      29373          1
         1     100000     262144
$ ls -al /var/spool/mail/
total 0
drwxrwxr-x. 1 root mail 30 Jul 29 20:00 .
drwxr-xr-x. 1 root root 68 Jul 29 20:00 ..
-rw-rw----. 1 1000 mail  0 Nov  7  2023 csb-inituser
-rw-rw----. 1 rpc  mail  0 Nov  7  2023 rpc

So I assume the issue is related to the mail user.

mail:x:8:12:mail:/var/spool/mail:/usr/sbin/nologin

[bfournie]

UPDATE: I realized I can by this by adding --storage-opt overlay.ignore_chown_errors=true on the inner podman command.

Which leads me the next issue that I need to investigate :-)

ime="2025-08-05T22:44:57Z" level=warning msg="Failed to add conmon to cgroupfs sandbox cgroup: creating cgroup path /libpod_parent/conmon: mkdir /sys/fs/cgroup/libpod_parent: permission denied"
Error: crun: create `/sys/fs/cgroup/libpod_parent`: Permission denied: OCI permission denied

So it would be good to know a more permanent solution that didn't require the overlay.ignore_chown_errors setting but that is fine for now.

Feel free to close this issue. Thank you.

[karampok]

Hi! I have reaced also this problem in my new laptop where user is created using homectl/systemd-homed (and therefore there is no traditional useradd). Have you find a more permanent solution?

[geoffroybr]

Did you manage to get this to work end-to-end ? I am facing a similar issue and I have already spent hours searching for a solution...

[bfournie]

@geoffroybr @karampok I wasn't able to get podman in podman working because of this and other permission issues. I ended up switching to buildah and got that to work with nested podman, although I did have to run it rootful. I'll stay with buildah since its more configurable and also consistent with the build system we're using, but I'm trying to simplify it and run it rootless. For reference, this is the buildah command that I got working - https://github.com/openshift/agent-installer-utils/blob/main/tools/iso_builder/Makefile#L31-L42