potentially insufficient UIDs or GIDs available in user namespace

[micah]

I am using mmdebstrap to build a Debian image from scratch. I'm doing that from within an unprivileged podman container running in a gitlab CI runner. The mmdebstrap command produces an uncompressed tarball on standard output. It succeeds without a problem, but when I pipe that tarball to podman import, I get the following error:

Error: writing blob: adding layer with blob "sha256:5f1469c71192378cbe23c8bd6d617bd0746f0c23d6ae6d092d0da02eb3c791b3": processing tar file(potentially insufficient UIDs or GIDs available in user namespace (requested 0:42 for /etc/gshadow): Check /etc/subuid and /etc/subgid if configured locally and run podman-system-migrate: lchown /etc/gshadow: invalid argument): exit status 1

The more specific debug log lines are:

time="2023-09-25T20:59:08Z" level=debug msg="Importing image from \"/var/tmp/podman3406765468\""
time="2023-09-25T20:59:09Z" level=debug msg="parsed reference into \"[overlay@/var/lib/containers/storage+/run/containers/storage]@59d1b2e60e8c0286a1411e8cc6e8ed09943acc73d8cbfdd035946235c25e3f57\""
time="2023-09-25T20:59:09Z" level=debug msg="Normalized platform linux/amd64 to {amd64 linux  [] }"
time="2023-09-25T20:59:09Z" level=debug msg="Copying source image /var/tmp/podman3406765468 to destination image [overlay@/var/lib/containers/storage+/run/containers/storage]@59d1b2e60e8c0286a1411e8cc6e8ed09943acc73d8cbfdd035946235c25e3f57"
time="2023-09-25T20:59:09Z" level=debug msg="Using blob info cache at /var/lib/containers/cache/blob-info-cache-v1.boltdb"
time="2023-09-25T20:59:09Z" level=debug msg="IsRunningImageAllowed for image tarball:"
time="2023-09-25T20:59:09Z" level=debug msg=" Using default policy section"
time="2023-09-25T20:59:09Z" level=debug msg=" Requirement 0: allowed"
time="2023-09-25T20:59:09Z" level=debug msg="Overall: allowed"
Getting image source signatures
time="2023-09-25T20:59:09Z" level=debug msg="Manifest has MIME type application/vnd.oci.image.manifest.v1+json, ordered candidate list [application/vnd.oci.image.manifest.v1+json, application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.v1+prettyjws, application/vnd.docker.distribution.manifest.v1+json]"
time="2023-09-25T20:59:09Z" level=debug msg="... will first try using the original manifest unmodified"
Copying blob sha256:f11b654799499dede1d3a58f89c4b977c0c42985da6a38ea7ef029fc133181b3
time="2023-09-25T20:59:09Z" level=debug msg="Checking if we can reuse blob sha256:f11b654799499dede1d3a58f89c4b977c0c42985da6a38ea7ef029fc133181b3: general substitution = true, compression for MIME type \"application/vnd.oci.image.layer.v1.tar\" = true"
time="2023-09-25T20:59:09Z" level=debug msg="No compression detected"
time="2023-09-25T20:59:09Z" level=debug msg="Using original blob without modification"
time="2023-09-25T20:59:11Z" level=debug msg="Applying tar in /var/lib/containers/storage/overlay/f11b654799499dede1d3a58f89c4b977c0c42985da6a38ea7ef029fc133181b3/diff"
Error: writing blob: adding layer with blob "sha256:f11b654799499dede1d3a58f89c4b977c0c42985da6a38ea7ef029fc133181b3": processing tar file(potentially insufficient UIDs or GIDs available in user namespace (requested 0:42 for /etc/gshadow): Check /etc/subuid and /etc/subgid if configured locally and run podman-system-migrate: lchown /etc/gshadow: invalid argument): exit status 1

I am able to do this on a privileged runner, without problems. Additionally, I can also build the image successfully with -storage-opt ignore_chown_errors=true to the podman import -- although I'm unclear what the consequences of this are. It is clearly a problem for running containers, but it seems like building is fine.

Some debug information:

# sudo -u gitlab-runner -i
gitlab-runner@ci-runner-x86-02:~$ podman info --debug
host:
  arch: amd64
  buildahVersion: 1.28.2
  cgroupControllers:
  - cpu
  - memory
  - pids
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon_2.1.6+ds1-1_amd64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.6, commit: unknown'
  cpuUtilization:
    idlePercent: 92.15
    systemPercent: 1.27
    userPercent: 6.59
  cpus: 16
  distribution:
    codename: bookworm
    distribution: debian
    version: "12"
  eventLogger: journald
  hostname: ci-runner-x86-02
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 997
      size: 1
    - container_id: 1
      host_id: 1000000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 999
      size: 1
    - container_id: 1
      host_id: 1000000
      size: 65536
  kernel: 6.1.0-12-amd64
  linkmode: dynamic
  logDriver: journald
  memFree: 253972078592
  memTotal: 316768120832
  networkBackend: cni
  ociRuntime:
    name: crun
    package: crun_1.8.1-1+b1_amd64
    path: /usr/bin/crun
    version: |-
      crun version 1.8.1
      commit: f8a096be060b22ccd3d5f3ebe44108517fbf6c30
      rundir: /run/user/999/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +YAJL
  os: linux
  remoteSocket:
    exists: true
    path: /run/user/999/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: false
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns_1.2.0-1_amd64
    version: |-
      slirp4netns version 1.2.0
      commit: 656041d45cfca7a4176f6b7eed9e4fe6c11e8383
      libslirp: 4.7.0
      SLIRP_CONFIG_VERSION_MAX: 4
      libseccomp: 2.5.4
  swapFree: 1073737728
  swapTotal: 1073737728
  uptime: 71h 59m 42.00s (Approximately 2.96 days)
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries:
  search:
  - docker.io
store:
  configFile: /home/gitlab-runner/.config/containers/storage.conf
  containerStore:
    number: 2
    paused: 0
    running: 1
    stopped: 1
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /home/gitlab-runner/.local/share/containers/storage
  graphRootAllocated: 315926315008
  graphRootUsed: 70429724672
  graphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 77
  runRoot: /run/user/999/containers
  volumePath: /home/gitlab-runner/.local/share/containers/storage/volumes
version:
  APIVersion: 4.3.1
  Built: 0
  BuiltTime: Thu Jan  1 00:00:00 1970
  GitCommit: ""
  GoVersion: go1.19.8
  Os: linux
  OsArch: linux/amd64
  Version: 4.3.1

root@ci-runner-x86-02:~# /sbin/getcap /usr/bin/newuidmap /usr/bin/newgidmap
root@ci-runner-x86-02:~# sudo -u gitlab-runner -i
gitlab-runner@ci-runner-x86-02:~$ podman system migrate
gitlab-runner@ci-runner-x86-02:~$ podman unshare cat /proc/self/uid_map
         0        999          1
         1    1000000      65536
gitlab-runner@ci-runner-x86-02:~$ podman unshare cat /proc/self/gid_map
         0        997          1
         1    1000000      65536
gitlab-runner@ci-runner-x86-02:~$ getsubids gitlab-runner
0: gitlab-runner 1000000 65536
gitlab-runner@ci-runner-x86-02:~$ getsubids -g gitlab-runner
0: gitlab-runner 1000000 65536
gitlab-runner@ci-runner-x86-02:~$ cat /etc/subuid 
gitlab-runner:1000000:65536
gitlab-runner@ci-runner-x86-02:~$ cat /etc/subgid
gitlab-runner:1000000:65536
gitlab-runner@ci-runner-x86-02:~$ id gitlab-runner
uid=999(gitlab-runner) gid=997(gitlab-runner) groups=997(gitlab-runner)

The newuidmap and newgidmap executables are available on the host as is slirp4netns, kernel.unprivileged_userns_clone = 1, and loginctl enable-linger gitlab-runner has been run. What is missing?

Here is the build script:

mmdebstrap \
  --mode=auto \
  --variant=minbase \
  --aptopt='Acquire::Languages "none"' \
  --dpkgopt='path-exclude=/usr/share/man/*' \
  --dpkgopt='path-exclude=/usr/share/man/man[1-9]/*' \
  --dpkgopt='path-exclude=/usr/share/locale/*' \
  --dpkgopt='path-include=/usr/share/locale/locale.alias' \
  --dpkgopt='path-exclude=/usr/share/lintian/*' \
  --dpkgopt='path-exclude=/usr/share/info/*' \
  --dpkgopt='path-exclude=/usr/share/doc/*' \
  --dpkgopt='path-include=/usr/share/doc/*/copyright' \
  --dpkgopt='path-exclude=/usr/share/omf/*' \
  --dpkgopt='path-exclude=/usr/share/help/*' \
  --dpkgopt='path-exclude=/usr/share/gnome/*' \
  --dpkgopt='path-exclude=/usr/share/examples/*' \
  --include='ca-certificates' \
 bookworm  - |  podman import -c 'CMD ["bash"]' - debian:bookworm-minbase

Running a strace -f on the podman import command, I see it fail here:

9652  newfstatat(AT_FDCWD, "/etc/gshadow", {st_mode=S_IFREG|0640, st_size=364, ...}, AT_SYMLINK_NOFOLLOW) = 0
9652  fchownat(AT_FDCWD, "/etc/gshadow", 0, 42, AT_SYMLINK_NOFOLLOW) = -1 EINVAL (Invalid argument)
9652  write(2, "potentially insufficient UIDs or"..., 225) = 225

Is podman import trying to activate user namespaces while it is in the docker container? Because this isn't a privileged container, where those are not setup in the container itself, would probably fail.

[micah]

As this discussion is now two weeks old, without a reply, I suspect that it either should have been filed as an issue, or nobody has any ideas of anything to try. I'll leave it open a little while longer, in case someone has an idea, but will close it later if there is no reply.

[bejean]

Hi @micah, did you find a solution to this problem ? We are experimenting the same issue on one of our podman server.
Regards.
Dominique

[micah]

Unfortunately, no... I did not get a solution to this.

[marcbull]

If you're having this issue on NixOS, just enable the following option to fix it:

https://search.nixos.org/options?channel=25.05&show=users.users.%3Cname%3E.autoSubUidGidRange&from=0&size=50&sort=relevance&type=packages&query=autoSubUidGidRange