FreeBSD run podman within JAIL(nested containers)

[PetruGarstea]

Greetings,

I have been trying to use podman in a long living jail container on FreeBSD 13.2 release.
The long living container is configured with linux emulation

zroot/bastille/jails/podman/root on / (zfs, local, noatime, nfsv4acls)
zroot/bastille/jails/podman/root/containers on /var/db/containers (zfs, local, noatime, nfsv4acls)
devfs on /compat/linux/dev (devfs)
tmpfs on /compat/linux/dev/shm (tmpfs, local)
fdescfs on /compat/linux/dev/fd (fdescfs)
linprocfs on /compat/linux/proc (linprocfs, local)
linsysfs on /compat/linux/sys (linsysfs, local)
/tmp on /compat/linux/tmp (nullfs, local, noatime, nosuid, nfsv4acls)
/usr/home on /compat/linux/home (nullfs, local, noatime, nfsv4acls)
/usr/local/bastille/releases/13.2-RELEASE on /.bastille (nullfs, local, noatime, read-only, nfsv4acls)
devfs on /dev (devfs)
fdescfs on /dev/fd (fdescfs)

ATM, I am trying to figure what configuration is missing on the system that causes:

Error pulling candidate docker.io/library/alpine:latest: copying system image from manifest list: writing blob: adding layer with blob "sha256:4abcf20661432fb2d719aaf90656f55c287f8ca915dc1c92ec14ff61e67fbaf8": ApplyLayer stdout: stderr: operation not permitted exit status 1

The full log

root@podman:~ # podman --log-level debug run --rm --os=linux docker://docker.io/alpine cat /etc/os-release
INFO[0000] podman filtering at log level debug
DEBU[0000] Called run.PersistentPreRunE(podman --log-level debug run --rm --os=linux docker://docker.io/alpine cat /etc/os-release)
DEBU[0000] Using conmon: "/usr/local/bin/conmon"
DEBU[0000] Initializing boltdb state at /var/db/containers/storage/libpod/bolt_state.db
DEBU[0000] Using graph driver zfs
DEBU[0000] Using graph root /var/db/containers/storage
DEBU[0000] Using run root /var/run/containers/storage
DEBU[0000] Using static dir /var/db/containers/storage/libpod
DEBU[0000] Using tmp dir /var/run/libpod
DEBU[0000] Using volume path /var/db/containers/storage/volumes
DEBU[0000] Using transient store: false
DEBU[0000] [graphdriver] trying provided driver "zfs"
DEBU[0000] ID:36a2c4c9-eeba-406a-b1e4-0da02dcc28be START /sbin/zfs list -rHp -t filesystem -o name,origin,used,available,mountpoint,compression,type,volsize,quota,referenced,written,logicalused,usedbydataset zroot/bastille/jails/podman/root/containers  storage-driver=zfs
DEBU[0000] ID:36a2c4c9-eeba-406a-b1e4-0da02dcc28be FINISH  storage-driver=zfs
DEBU[0000] Initializing event backend file
DEBU[0000] Configured OCI runtime youki initialization failed: no valid executable found for OCI runtime youki: invalid argument
DEBU[0000] Configured OCI runtime krun initialization failed: no valid executable found for OCI runtime krun: invalid argument
DEBU[0000] Configured OCI runtime crun-wasm initialization failed: no valid executable found for OCI runtime crun-wasm: invalid argument
DEBU[0000] Configured OCI runtime runc initialization failed: no valid executable found for OCI runtime runc: invalid argument
DEBU[0000] Configured OCI runtime kata initialization failed: no valid executable found for OCI runtime kata: invalid argument
DEBU[0000] Configured OCI runtime runsc initialization failed: no valid executable found for OCI runtime runsc: invalid argument
DEBU[0000] Configured OCI runtime crun initialization failed: no valid executable found for OCI runtime crun: invalid argument
DEBU[0000] Configured OCI runtime runj initialization failed: no valid executable found for OCI runtime runj: invalid argument
DEBU[0000] Using OCI runtime "/usr/local/bin/ocijail"
INFO[0000] Setting parallel job count to 13
DEBU[0000] Successfully loaded 1 networks
DEBU[0000] Pulling image docker://docker.io/alpine (policy: missing)
DEBU[0000] Looking up image "docker.io/library/alpine:latest" in local containers storage
DEBU[0000] Normalized platform linux/amd64 to {amd64 linux  [] }
DEBU[0000] Trying "docker.io/library/alpine:latest" ...
DEBU[0000] reference "[zfs@/var/db/containers/storage+/var/run/containers/storage]docker.io/library/alpine:latest" does not resolve to an image ID
DEBU[0000] Trying "docker.io/library/alpine:latest" ...
DEBU[0000] reference "[zfs@/var/db/containers/storage+/var/run/containers/storage]docker.io/library/alpine:latest" does not resolve to an image ID
DEBU[0000] Trying "docker.io/library/alpine:latest" ...
DEBU[0000] Enforcing pull policy to "newer" to pull custom platform (arch: "", os: "linux", variant: "") - local image may mistakenly specify wrong platform
DEBU[0000] Loading registries configuration "/usr/local/etc/containers/registries.conf"
DEBU[0000] Normalized platform linux/amd64 to {amd64 linux  [] }
DEBU[0000] Attempting to pull candidate docker.io/library/alpine:latest for docker.io/library/alpine:latest
DEBU[0000] parsed reference into "[zfs@/var/db/containers/storage+/var/run/containers/storage]docker.io/library/alpine:latest"
Trying to pull docker.io/library/alpine:latest...
DEBU[0000] Copying source image //alpine:latest to destination image [zfs@/var/db/containers/storage+/var/run/containers/storage]docker.io/library/alpine:latest
DEBU[0000] Using registries.d directory /usr/local/etc/containers/registries.d
DEBU[0000] Trying to access "docker.io/library/alpine:latest"
DEBU[0000] No credentials matching docker.io/library/alpine found in /root/.config/containers/auth.json
DEBU[0000] No credentials matching docker.io/library/alpine found in /root/.config/containers/auth.json
DEBU[0000] No credentials matching docker.io/library/alpine found in /root/.docker/config.json
DEBU[0000] No credentials matching docker.io/library/alpine found in /root/.dockercfg
DEBU[0000] No credentials for docker.io/library/alpine found
DEBU[0000]  No signature storage configuration found for docker.io/library/alpine:latest, using built-in default file:///var/lib/containers/sigstore
DEBU[0000] Looking for TLS certificates and private keys in /usr/local/etc/docker/certs.d/docker.io
DEBU[0000] GET https://registry-1.docker.io/v2/
DEBU[0000] Ping https://registry-1.docker.io/v2/ status 401
DEBU[0000] GET https://auth.docker.io/token?scope=repository%3Alibrary%2Falpine%3Apull&service=registry.docker.io
DEBU[0000] GET https://registry-1.docker.io/v2/library/alpine/manifests/latest
DEBU[0000] Content-Type from manifest GET is "application/vnd.docker.distribution.manifest.list.v2+json"
DEBU[0000] Using SQLite blob info cache at /var/lib/containers/cache/blob-info-cache-v1.sqlite
DEBU[0000] Source is a manifest list; copying (only) instance sha256:6457d53fb065d6f250e1504b9bc42d5b6c65941d57532c072d929dd0628977d0 for current system
DEBU[0000] GET https://registry-1.docker.io/v2/library/alpine/manifests/sha256:6457d53fb065d6f250e1504b9bc42d5b6c65941d57532c072d929dd0628977d0
DEBU[0000] Content-Type from manifest GET is "application/vnd.docker.distribution.manifest.v2+json"
DEBU[0000] IsRunningImageAllowed for image docker:docker.io/library/alpine:latest
DEBU[0000]  Using default policy section
DEBU[0000]  Requirement 0: allowed
DEBU[0000] Overall: allowed
DEBU[0000] Downloading /v2/library/alpine/blobs/sha256:05455a08881ea9cf0e752bc48e61bbd71a34c029bb13df01e40e3e70e0d007bd
DEBU[0000] GET https://registry-1.docker.io/v2/library/alpine/blobs/sha256:05455a08881ea9cf0e752bc48e61bbd71a34c029bb13df01e40e3e70e0d007bd
Getting image source signatures
DEBU[0000] Reading /var/lib/containers/sigstore/library/alpine@sha256=6457d53fb065d6f250e1504b9bc42d5b6c65941d57532c072d929dd0628977d0/signature-1
DEBU[0000] Not looking for sigstore attachments: disabled by configuration
DEBU[0000] Manifest has MIME type application/vnd.docker.distribution.manifest.v2+json, ordered candidate list [application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.v1+prettyjws, application/vnd.oci.image.manifest.v1+json, application/vnd.docker.distribution.manifest.v1+json]
DEBU[0000] ... will first try using the original manifest unmodified
DEBU[0000] Checking if we can reuse blob sha256:4abcf20661432fb2d719aaf90656f55c287f8ca915dc1c92ec14ff61e67fbaf8: general substitution = true, compression for MIME type "application/vnd.docker.image.rootfs.diff.tar.gzip" = true
DEBU[0000] Failed to retrieve partial blob: format not supported on this system
DEBU[0000] Downloading /v2/library/alpine/blobs/sha256:4abcf20661432fb2d719aaf90656f55c287f8ca915dc1c92ec14ff61e67fbaf8
DEBU[0000] GET https://registry-1.docker.io/v2/library/alpine/blobs/sha256:4abcf20661432fb2d719aaf90656f55c287f8ca915dc1c92ec14ff61e67fbaf8
Copying blob 4abcf2066143 [--------------------------------------] 0.0b / 3.3MiB (skipped: 0.0b = 0.00%)
Copying blob 4abcf2066143 [--------------------------------------] 0.0b / 3.3MiB | 0.0 b/s
Copying blob 4abcf2066143 done   |
Copying blob 4abcf2066143 done   |
DEBU[0001] ID:62d93b96-1b16-4703-8999-a2ba584f1bc5 FINISH  storage-driver=zfs
DEBU[0001] ID:1871d56d-a96a-4a0d-8355-6688f206d776 START /sbin/zfs list -Hp -o name,origin,used,available,mountpoint,compression,type,volsize,quota,referenced,written,logicalused,usedbydataset zroot/bastille/jails/podman/root/containers/d4fc045c9e3a848011de66f34b81f052d4f2c15a17bb196d637e526349601820  storage-driver=zfs
Copying blob 4abcf2066143 done   |
DEBU[0001] mount("zroot/bastille/jails/podman/root/containers/d4fc045c9e3a848011de66f34b81f052d4f2c15a17bb196d637e526349601820", "/var/db/containers/storage/zfs/graph/d4fc045c9e3a848011de66f34b81f052d4f2c15a17bb196d637e526349601820", "")  storage-driver=zfs
DEBU[0001] Start untar layer
ERRO[0001] While applying layer: ApplyLayer stdout:  stderr: operation not permitted exit status 1
DEBU[0001] unmount("/var/db/containers/storage/zfs/graph/d4fc045c9e3a848011de66f34b81f052d4f2c15a17bb196d637e526349601820")  storage-driver=zfs
DEBU[0001] ID:acefec41-353b-4871-a2e7-a60a7b239d94 START /sbin/zfs destroy -r zroot/bastille/jails/podman/root/containers/d4fc045c9e3a848011de66f34b81f052d4f2c15a17bb196d637e526349601820  storage-driver=zfs
DEBU[0001] ID:acefec41-353b-4871-a2e7-a60a7b239d94 FINISH  storage-driver=zfs
DEBU[0001] Error pulling candidate docker.io/library/alpine:latest: copying system image from manifest list: writing blob: adding layer with blob "sha256:4abcf20661432fb2d719aaf90656f55c287f8ca915dc1c92ec14ff61e67fbaf8": ApplyLayer stdout:  stderr: operation not permitted exit status 1
Error: copying system image from manifest list: writing blob: adding layer with blob "sha256:4abcf20661432fb2d719aaf90656f55c287f8ca915dc1c92ec14ff61e67fbaf8": ApplyLayer stdout:  stderr: operation not permitted exit status 1
DEBU[0001] Shutting down engines

cc @dfr

Thanks in advance,
Petru