Nesting containers suddenly fails although enough subuid's at nesting level 3...

[gabyx]

I wanted to test how many times we can nest containers (podman images)
I use the following Containerfile:

FROM quay.io/podman/stable as base

COPY ./Containerfile ./run.sh /home/podman/
RUN chmod +x /home/podman/*.sh

WORKDIR /home/podman

and the run.sh:

#!/usr/bin/env bash
#
set -u
# set -e

DIR=$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" &>/dev/null && pwd)
cd "$DIR" || exit 1

level="${1:-1}"

# Build the image.
echo ">>>> $level. Container: inside"

# Run the image and build again.
if [ "$level" -lt 10 ]; then
    echo ">>>> $level. Container: Launching a new container ..."
    podman run -u podman --rm ttl.sh/podman-test ./run.sh "$((level + 1))"
else
    echo ">>>> $level. Container: Finally reached container level: $level"
fi

echo ">>>> $level. Container: leaving"

and the I run the recursion with:

name=ttl.sh/podman-test
podman volume create podman-vol
podman build -f Containerfile -t "$name" .
podman push "$name"

podman run --privileged --rm  -it "$name" ./run.sh

I have the followin in /etc/subuids and /etc/subgids:

nixos:100000:1000000

which means I can allocate 1000000 subuid's etc. which should be enough to run the container nest recursion to level above 3:
the output however crashes with:

>>>> 1. Container: inside
>>>> 1. Container: Launching a new container ...
...
Writing manifest to image destination
>>>> 2. Container: inside
>>>> 2. Container: Launching a new container ...
Trying to pull ttl.sh/podman-test:latest...
Getting image source signatures
Copying blob sha256:58392155411eb9d283ae42996708712e584dd2f4b1ccb05fcb2f70da6bb126b7
Copying blob sha256:e5e7cb16f89793cae00a828afb144d1bc1ad593a1c23d98845d6fc16a2a96ec4
...
Writing manifest to image destination
>>>> 3. Container: inside
>>>> 3. Container: Launching a new container ...
time="2024-03-14T18:59:22Z" level=error msg="running `/usr/bin/newuidmap 21 0 1000 1 1 1 999 1000 1001 64535`: newuidmap: Target process is owned by a different user: uid:1000 pw_uid:1000 st_uid:0, gid:1000 pw_gid:1000 st_gid:0\n"
Error: cannot set up namespace using "/usr/bin/newuidmap": exit status 1
>>>> 3. Container: leaving
>>>> 2. Container: leaving
>>>> 1. Container: leaving

Anybody has an idea what the problem is, that after nesting level 2, the recursion gives up??

Its an intersting thing to nest containers, better know as "podman in podman" etc frequently arising in CI etc.

This is nice if you containerize the build system but don't want to pack everything into one large image nor want to divide it into stages called by some main script on the host.

Thanks for any insight.

[eriksjolund]

Nice initiative. Let's explore this :)
This is not feedback on your script but may be interesting none the less:

There is a Hacker News comment from user Klasiaster about running Podman in Podman in Podman.
See Hacker News comment 17 April 2022

---

Another thing: I wrote a Hacker News comment 26 April 2023 about
passing a file descriptor over nested Podman

The new systemd directive OpenFile= opens up the possibility to pass the file descriptor of a file from the host to a container running in a container.

sudo systemd-run --property User=test --property OpenFile=/etc/secretfile.txt \
                   --collect --pipe --wait --quiet \
          podman run --security-opt label=disable --user podman --device /dev/fuse quay.io/podman/stable \
          podman run -q alpine sh -c "cat <&3"

[giuseppe]

could it be that the storage volume is mounted with nosuid so newuidmap won't work in the nested environment?

You'd probably need to pass the volume manually and make sure each sub-level gets a distinct directory

[gabyx]

What do you mean the storage volume: Do you mean the --runroot and --root arguments (defaulting to ~/.local/share/containers/...) etc?

I also try now to increase the ulimit -a for nproc and nofile, maybe thats a resource problem...
Each new podman instance IMO uses the default directories afaik when I do:
podman run -u podman --rm ttl.sh/podman-test ./run.sh "$((level + 1))"

[gabyx]

ulimit increase did not help.

[giuseppe]

yes, you need to make sure they are not on a volume that has nosuid set

[gabyx]

hm... ok. I start the recursion with podman on the system using my local ~/.local/share/podman etc...
mount reports:

/dev/mapper/enc-physical-vol on /home type btrfs (rw,noatime,compress=zstd:3,ssd,space_cache=v2,subvolid=257,subvol=/home)
so that should be ok. Also the first podman-in-podman (2. Container: inside) level works, but fails on the next one?

[gabyx]

The following works till level 5 and more...
I added a podman volume create storage at each level in each podman in podman:

The following:

# Original image.
FROM quay.io/podman/stable as original
COPY ./Containerfile ./run.sh /home/podman/
RUN chmod +x /home/podman/*.sh

WORKDIR /home/podman

podman build -f Containerfile -t "$name" --target original .
podman push "$name"

podman volume rm podman-root && podman volume create podman-root || true
podman run --privileged --device /dev/fuse -v "podman-root:/podman-root" -v "$HOME/.local/share/containers/storage:/var/lib/shared" --rm "$name" ./run.sh

sr/bin/env bash
#
set -u
set -e

DIR=$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" &>/dev/null && pwd)
cd "$DIR" || exit 1

level="${1:-1}"
vol_name="podman-root-$level"
msg="-> $level. Container"

function indent() {
    cat | sed "s@^@| @g"
}

function run_podman() {
    # When you specify `--root` , than `storage-opts`
    # in `/etc/containers/storage.conf` are ignored.
    podman \
        --root /podman-root/root \
        --runroot /podman-root/runroot \
        --storage-opt "additionalimagestore=/var/lib/shared" \
        "$@"
}

function main() {
    # Build the image.
    echo "$msg: inside [version: $(cat /image.version)]"

    # Run the image and build again.
    if [ "$level" -lt 5 ]; then
        echo "$msg: Launching a new container ..."

        # We need to make a new volume for the next podman
        # to have the stuff it needs stored.
        echo "$msg: create volume:"
        run_podman volume create "$vol_name" || {
            echo "create failed: $vol_name"
            exit 3
        }

        # We launch the new podman with root/runroot
        # on the current mounted volume `data`.
        # Then we mount the current data as
        # [`additionalimages`](https://www.redhat.com/sysadmin/image-stores-podman)
        # to next podman to have caching.

        run_podman \
            run \
            -v "$vol_name:/podman-root:Z" \
            -v "/var/lib/shared:/var/lib/shared" \
            --privileged \
            --rm ttl.sh/podman-test \
            ./run.sh "$((level + 1))" || true

        echo

    else
        echo "$msg: Finally reached container level: $level"
    fi

    echo "$msg: leaving"
}

main 2>&1 | indent

Questions:

• So now the question resides why I need to create a volume for each level where podman will store its stuff:
• Can we share the downloaded containers somehow over the levels? So have to enable some caching here?

[giuseppe]

• Can we share the downloaded containers somehow over the levels? So have to enable some caching here?

mount the current storage as an additional images store in the nested container

[gabyx]

Thanks I added the -v "/storage:/var/lib/shared mount to the last post, and it works crazy:

| -> 1. Container: inside [version: a78646a804879c2c8e9fbafa6bce0c12]
| -> 1. Container: Launching a new container ...
| -> 1. Container: create volume:
| podman-root-1
| time="2024-03-19T10:09:33Z" level=warning msg="Path \"/run/secrets/etc-pki-entitlement\" from \"/etc/containers/mounts.conf\" doesn't exist, skipping"
| time="2024-03-19T10:09:33Z" level=warning msg="Path \"/run/secrets/rhsm\" from \"/etc/containers/mounts.conf\" doesn't exist, skipping"
| | -> 2. Container: inside [version: a78646a804879c2c8e9fbafa6bce0c12]
| | -> 2. Container: Launching a new container ...
| | -> 2. Container: create volume:
| | podman-root-2
| | time="2024-03-19T10:09:33Z" level=warning msg="Path \"/run/secrets/etc-pki-entitlement\" from \"/etc/containers/mounts.conf\" doesn't exist, skipping"
| | time="2024-03-19T10:09:33Z" level=warning msg="Path \"/run/secrets/rhsm\" from \"/etc/containers/mounts.conf\" doesn't exist, skipping"
| | | -> 3. Container: inside [version: a78646a804879c2c8e9fbafa6bce0c12]
| | | -> 3. Container: Launching a new container ...
| | | -> 3. Container: create volume:
| | | podman-root-3
| | | time="2024-03-19T10:09:33Z" level=warning msg="Path \"/run/secrets/etc-pki-entitlement\" from \"/etc/containers/mounts.conf\" doesn't exist, skipping"
| | | time="2024-03-19T10:09:33Z" level=warning msg="Path \"/run/secrets/rhsm\" from \"/etc/containers/mounts.conf\" doesn't exist, skipping"
| | | | -> 4. Container: inside [version: a78646a804879c2c8e9fbafa6bce0c12]
| | | | -> 4. Container: Launching a new container ...
| | | | -> 4. Container: create volume:
| | | | podman-root-4
| | | | time="2024-03-19T10:09:33Z" level=warning msg="Path \"/run/secrets/etc-pki-entitlement\" from \"/etc/containers/mounts.conf\" doesn't exist, skipping"
| | | | time="2024-03-19T10:09:33Z" level=warning msg="Path \"/run/secrets/rhsm\" from \"/etc/containers/mounts.conf\" doesn't exist, skipping"
| | | | | -> 5. Container: inside [version: a78646a804879c2c8e9fbafa6bce0c12]
| | | | | -> 5. Container: Finally reached container level: 5
| | | | | -> 5. Container: leaving
| | | | 
| | | | -> 4. Container: leaving
| | | 
| | | -> 3. Container: leaving
| | 
| | -> 2. Container: leaving
| 
| -> 1. Container: leaving

[gabyx]

But that is only with running as root with user podman I get some other troubles I need to resolve...

[gabyx]

From issue: #22088: To continue the discussion here:

Discovering the possibility to nest containers in https://github.com/gabyx/container-nesting (which was specifically made for this bug report to properly test this).
This repo uses the podman image to nest containers, so doing podman-in-podman-in-podman-.....

Note: This container nesting is not some academic example (although for nesting levels >2 it migth be), this usecase appears in CI when we want the ability to not have everything in one top-level image only but leave room to run other containers inside the top-level one.

Container nesting works with user root.

It seems that when running the container nesting recursion as user podman together with --userns=keep-id:uid=1000,gid=1000 something is at odd when looking at the permission how the files are mounted at nesting level >1 (here only /home/podman is relevant but other directories have similar issues).

I dont think its a podman issue (hopefully) but maybe a image issue or a configuration issue. It would however be nice if the podman image would support such kind of nesting (1-2 levels).

Steps to reproduce the issue

Steps to reproduce the issue

1. git clone --branch bugfix/report-bug-podman https://github.com/gabyx/container-nesting

2. cd container-nesting

3. Check that the root recursion works:

   just podman custom root

   or

   cd src/podman && start-run.sh custom root false

   The output should be

     ===========================================================
     ===========================================================
     ==================== Start Recursion ======================
     ===========================================================
     ===========================================================
    | -> 1. Container: inside [version: 36ff37d9a63c55f5974889fcc8b778c4]
    | -> 1. Container: Launching a new container ...
    | -> 1. Container: create volume:
    | podman-root-1
    | -> 1. Container: Simple container run test:
    | NAME="Alpine Linux"
    | -> 1. Container: Start new container:
    |
    | | -> 2. Container: inside [version: 36ff37d9a63c55f5974889fcc8b778c4]
    | | -> 2. Container: Launching a new container ...
    | | -> 2. Container: create volume:
    | | podman-root-2
    | | -> 2. Container: Simple container run test:
    | | NAME="Alpine Linux"
    | | -> 2. Container: Start new container:
    | |
    | | | -> 3. Container: inside [version: 36ff37d9a63c55f5974889fcc8b778c4]
    | | | -> 3. Container: Launching a new container ...
    | | | -> 3. Container: create volume:
    | | | podman-root-3
    | | | -> 3. Container: Simple container run test:
    | | | NAME="Alpine Linux"
    | | | -> 3. Container: Start new container:
    | | |
    | | | | -> 4. Container: inside [version: 36ff37d9a63c55f5974889fcc8b778c4]
    | | | | -> 4. Container: Launching a new container ...
    | | | | -> 4. Container: create volume:
    | | | | podman-root-4
    | | | | -> 4. Container: Simple container run test:
    | | | | NAME="Alpine Linux"
    | | | | -> 4. Container: Start new container:
    | | | |
    | | | | | -> 5. Container: inside [version: 36ff37d9a63c55f5974889fcc8b778c4]
    | | | | | -> 5. Container: Finally reached container level: 5
    | | | | | -> 5. Container: leaving
    | | | |
    | | | | -> 4. Container: leaving
    | | |
    | | | -> 3. Container: leaving
    | |
    | | -> 2. Container: leaving
    |
    | -> 1. Container: leaving
    

4. Run the recursion with user podman.
   This will use --userns=keepid:uid=1000,gid=1000 to make all mounts use podman user.

   cd src/podman && start-run.sh custom podman false

   This will crash. To inspect whats going on use start-run.sh custom podman true which will put you in a bash shell right after entering every container and right after leaving. Exit the shell to continue the recursion.

5. Inspect inside the -> 1. Container: entering that
   ls -alnd /home/podman has permission 1000.

6. Inspect inside the -> 2. Container: entering that
   ls -aln /home/podman has not permission 1000 but on my machine its 999 which is weird.

Note: The image which is used is an alpine image with podman installed. You can also use original instead of custom above to build and use the original quay.io/podman/stable image. The findings are the same.

Describe the results you received

The permissions on ls -aln /home/podman result in 1000 in the 1. container and not 1000 in the 2. container.
when podman run --userns=keepid:uid=1000,gid=1000 ... is launched from within the 1. container.

Describe the results you expected

The /home/podman directory should be mounted with permissions 1000 also in the 2. container and on further deeper levels.

Note: I am using --privileged all the time, not sure if thats wrong...

podman info output

host:
  arch: amd64
  buildahVersion: 1.33.5
  cgroupControllers:
  - cpu
  - io
  - memory
  - pids
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: Unknown
    path: /nix/store/wp1wic4bsd28qb7hs04dr7dsgls5j8pf-conmon-2.1.10/bin/conmon
    version: 'conmon version 2.1.10, commit: '
  cpuUtilization:
    idlePercent: 98.82
    systemPercent: 0.58
    userPercent: 0.6
  cpus: 32
  databaseBackend: sqlite
  distribution:
    codename: uakari
    distribution: nixos
    version: "24.05"
  eventLogger: journald
  freeLocks: 2019
  hostname: linux-nixos
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 100
      size: 1
    - container_id: 1
      host_id: 100000
      size: 10000000
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 10000000
  kernel: 6.6.19
  linkmode: dynamic
  logDriver: journald
  memFree: 39401140224
  memTotal: 67342225408
  networkBackend: netavark
  networkBackendInfo:
    backend: netavark
    dns:
      package: Unknown
      path: /nix/store/vwrqp3pfj17p0zd5gdnrkrvr702z838r-podman-4.9.3/libexec/podman/aardvark-dns
      version: aardvark-dns 1.10.0
    package: Unknown
    path: /nix/store/vwrqp3pfj17p0zd5gdnrkrvr702z838r-podman-4.9.3/libexec/podman/netavark
    version: netavark 1.7.0
  ociRuntime:
    name: crun
    package: Unknown
    path: /nix/store/h9wpwj1zjz6qjw3q331k757cq27kmj9k-crun-1.14.4/bin/crun
    version: |-
      crun version 1.14.4
      commit: 1.14.4
      rundir: /run/user/1000/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +YAJL
  os: linux
  pasta:
    executable: ""
    package: ""
    version: ""
  remoteSocket:
    exists: true
    path: /run/user/1000/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: ""
    selinuxEnabled: false
  serviceIsRemote: false
  slirp4netns:
    executable: /nix/store/vwrqp3pfj17p0zd5gdnrkrvr702z838r-podman-4.9.3/libexec/podman/slirp4netns
    package: Unknown
    version: |-
      slirp4netns version 1.2.3
      commit: c22fde291bb35b354e6ca44d13be181c76a0a432
      libslirp: 4.7.0
      SLIRP_CONFIG_VERSION_MAX: 4
      libseccomp: 2.5.5
  swapFree: 77309407232
  swapTotal: 77309407232
  uptime: 5h 53m 35.00s (Approximately 0.21 days)
  variant: ""
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries:
  search:
  - docker.io
  - quay.io
store:
  configFile: /home/nixos/.config/containers/storage.conf
  containerStore:
    number: 25
    paused: 0
    running: 0
    stopped: 25
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /home/nixos/.local/share/containers/storage
  graphRootAllocated: 1049659179008
  graphRootUsed: 181234888704
  graphStatus:
    Backing Filesystem: btrfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Supports shifting: "false"
    Supports volatile: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 622
  runRoot: /run/user/1000/containers
  transientStore: false
  volumePath: /home/nixos/.local/share/containers/storage/volumes
version:
  APIVersion: 4.9.3
  Built: 315532800
  BuiltTime: Tue Jan  1 01:00:00 1980
  GitCommit: ""
  GoVersion: go1.21.7
  Os: linux
  OsArch: linux/amd64
  Version: 4.9.3

[giuseppe]

you need to do it manually chmod -R 755 /path/to/the/store

[gabyx]

This seems the same issue: #14017
Still unsolved??

[gabyx]

So that means this chmod needs to happen also in nested containers (using fuse-overlayfs) when new images are pulled?

[gabyx]

You mean something like this:

sudo find ~/.local/share/containers/storage -type f | xargs -P0 -I {} sudo chmod go+r {} && \
sudo find ~/.local/share/containers/storage -type d | xargs -P0 -I {} sudo chmod go+rx {}

[gabyx]

As soon as I start the first container after I changed the permissions as you said on the host
I get weird behavior podman info inside the first container

ERRO[0000] mkdir /home/podman/rundir/libpod: no such file or directory
ls -alnd ~
049bba4fa32a:~$ ls -alnd ~
drwxr-xr-x    4 0        0               38 Mar 21 16:24 /home/podman

which now mounts the /home/podman as root ??

I did podman system reset before recreating the storage.