What's the expected configuration for shared image storage?

[addsoa]

I want to configure podman to provide a shared image storage between root-less users within the host but failed. I don't care if root user works.

user0 is sudo user, user1 and user2 are root-less users. All shells are ssh-ed with corresponding username directly (i.e. not by su).

• user1 output

user1@test:~$ podman images
REPOSITORY  TAG         IMAGE ID    CREATED     SIZE
user1@test:~$ podman pull busybox
Resolving "busybox" using unqualified-search registries (/etc/containers/registries.conf)
Trying to pull gitea.cn:666/busybox:latest...
Getting image source signatures
Copying blob a10d77880eaf done
Copying config 87ff76f62d done
Writing manifest to image destination
Storing signatures
87ff76f62d367950186bde563642e39208c0e2b4afc833b4b3b01b8fef60ae9e

user1@test:~$ podman images --log-level=debug
INFO[0000] podman filtering at log level debug
DEBU[0000] Called images.PersistentPreRunE(podman images --log-level=debug)
DEBU[0000] overlay storage already configured with a mount-program
DEBU[0000] Merged system config "/usr/share/containers/containers.conf"
DEBU[0000] overlay storage already configured with a mount-program
DEBU[0000] Using conmon: "/usr/bin/conmon"
DEBU[0000] Initializing boltdb state at /var/lib/sharedimages/libpod/bolt_state.db
DEBU[0000] Overriding run root "/run/user/1001" with "/run/user/1001/containers" from database
DEBU[0000] systemd-logind: Unknown object '/'.
DEBU[0000] Using graph driver overlay
DEBU[0000] Using graph root /var/lib/sharedimages
DEBU[0000] Using run root /run/user/1001/containers
DEBU[0000] Using static dir /var/lib/sharedimages/libpod
DEBU[0000] Using tmp dir /run/user/1001/libpod/tmp
DEBU[0000] Using volume path /var/lib/sharedimages/volumes
DEBU[0000] overlay storage already configured with a mount-program
DEBU[0000] Set libpod namespace to ""
DEBU[0000] [graphdriver] trying provided driver "overlay"
DEBU[0000] cached value indicated that overlay is supported
DEBU[0000] cached value indicated that metacopy is not being used
DEBU[0000] cached value indicated that native-diff is usable
DEBU[0000] backingFs=extfs, projectQuotaSupported=false, useNativeDiff=true, usingMetacopy=false
DEBU[0000] Initializing event backend journald
DEBU[0000] configured OCI runtime runc initialization failed: no valid executable found for OCI runtime runc: invalid argument
DEBU[0000] configured OCI runtime kata initialization failed: no valid executable found for OCI runtime kata: invalid argument
DEBU[0000] configured OCI runtime runsc initialization failed: no valid executable found for OCI runtime runsc: invalid argument
DEBU[0000] Using OCI runtime "/usr/bin/crun"
INFO[0000] Found CNI network podman (type=bridge) at /home/user1/.config/cni/net.d/87-podman.conflist
DEBU[0000] Default CNI network name podman is unchangeable
INFO[0000] Setting parallel job count to 37
DEBU[0000] parsed reference into "[overlay@/var/lib/sharedimages+/run/user/1001/containers]@87ff76f62d367950186bde563642e39208c0e2b4afc833b4b3b01b8fef60ae9e"
DEBU[0000] exporting opaque data as blob "sha256:87ff76f62d367950186bde563642e39208c0e2b4afc833b4b3b01b8fef60ae9e"
DEBU[0000] exporting opaque data as blob "sha256:87ff76f62d367950186bde563642e39208c0e2b4afc833b4b3b01b8fef60ae9e"
REPOSITORY                                TAG         IMAGE ID      CREATED        SIZE
gitea.cn:666/busybox  latest      87ff76f62d36  16 months ago  4.5 MB
DEBU[0000] Called images.PersistentPostRunE(podman images --log-level=debug)

user1@test:~$

• user2 output

user2@test:~$ podman images --log-level=debug
INFO[0000] podman filtering at log level debug
DEBU[0000] Called images.PersistentPreRunE(podman images --log-level=debug)
DEBU[0000] Merged system config "/usr/share/containers/containers.conf"
DEBU[0000] Using conmon: "/usr/bin/conmon"
DEBU[0000] Initializing boltdb state at /var/lib/sharedimages/libpod/bolt_state.db
DEBU[0000] Overriding run root "/run/user/1002" with "/run/user/1001/containers" from database
DEBU[0000] Overriding tmp dir "/run/user/1002/libpod/tmp" with "/run/user/1001/libpod/tmp" from database
DEBU[0000] systemd-logind: Unknown object '/'.
WARN[0000] XDG_RUNTIME_DIR is pointing to a path which is not writable. Most likely podman will fail.
DEBU[0000] Using graph driver overlay
DEBU[0000] Using graph root /var/lib/sharedimages
DEBU[0000] Using run root /run/user/1001/containers
DEBU[0000] Using static dir /var/lib/sharedimages/libpod
DEBU[0000] Using tmp dir /run/user/1001/libpod/tmp
DEBU[0000] Using volume path /var/lib/sharedimages/volumes
DEBU[0000] Set libpod namespace to ""
DEBU[0000] Not configuring container store
DEBU[0000] Initializing event backend journald
Error: error creating tmpdir: mkdir /run/user/1001/libpod: permission denied
user2@test:~$

The configurations:

• user1

user1@test:~$ more /home/user1/.config/containers/storage.conf
[storage]
driver = "overlay"
runroot = "/run/user/1001"
graphroot = "/var/lib/sharedimages"

user1@test:~$ id
uid=1001(user1) gid=1001(user1) groups=1001(user1)
user1@test:~$

• user2

user2@test:~$ more /home/user2/.config/containers/storage.conf
[storage]
driver = "overlay"
runroot = "/run/user/1002"
graphroot = "/var/lib/sharedimages"
[storage.options]
mount_program = "/bin/fuse-overlayfs"
user2@test:~$ id
uid=1002(user2) gid=1002(user2) groups=1002(user2)

• user0: no configuration

user0@test:/home/user1$ cd /etc/containers/
user0@test:/etc/containers$ ls
libpod.conf  policy.json  registries.conf  registries.conf.d
user0@test:/etc/containers$
user0@test:/etc/containers$ id
uid=1000(user0) gid=1000(user0) groups=1000(user0),4(adm),24(cdrom),27(sudo),30(dip),46(plugdev),110(lxd)
user0@test:/etc/containers$

• The podman version

user1@test:~$ podman version
Version:      3.4.4
API Version:  3.4.4
Go Version:   go1.18.1
Built:        Thu Jan  1 00:00:00 1970
OS/Arch:      linux/amd64

• The podman info

user1@test:~$ podman info
host:
  arch: amd64
  buildahVersion: 1.23.1
  cgroupControllers:
  - memory
  - pids
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: 'conmon: /usr/bin/conmon'
    path: /usr/bin/conmon
    version: 'conmon version 2.0.25, commit: unknown'
  cpus: 12
  distribution:
    codename: jammy
    distribution: ubuntu
    version: "22.04"
  eventLogger: journald
  hostname: test
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1001
      size: 1
    - container_id: 1
      host_id: 165536
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1001
      size: 1
    - container_id: 1
      host_id: 165536
      size: 65536
  kernel: 5.15.0-122-generic
  linkmode: dynamic
  logDriver: journald
  memFree: 19415232512
  memTotal: 20680478720
  ociRuntime:
    name: crun
    package: 'crun: /usr/bin/crun'
    path: /usr/bin/crun
    version: |-
      crun version 0.17
      commit: 0e9229ae34caaebcb86f1fde18de3acaf18c6d9a
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +YAJL
  os: linux
  remoteSocket:
    exists: true
    path: /run/user/1001/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: false
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: 'slirp4netns: /usr/bin/slirp4netns'
    version: |-
      slirp4netns version 1.0.1
      commit: 6a7b16babc95b6a3056b33fb45b74a6f62262dd4
      liuser0irp: 4.6.1
  swapFree: 8589930496
  swapTotal: 8589930496
  uptime: 13h 8m 55.68s (Approximately 0.54 days)
plugins:
  log:
  - k8s-file
  - none
  - journald
  network:
  - bridge
  - macvlan
  volume:
  - local
registries:
  search:
  - gitea.cn:666
store:
  configFile: /home/user1/.config/containers/storage.conf
  containerStore:
    number: 0
    paused: 0
    running: 0
    stopped: 0
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /var/lib/sharedimages
  graphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Using metacopy: "false"
  imageStore:
    number: 1
  runRoot: /run/user/1001/containers
  volumePath: /var/lib/sharedimages/volumes
version:
  APIVersion: 3.4.4
  Built: 0
  BuiltTime: Thu Jan  1 00:00:00 1970
  GitCommit: ""
  GoVersion: go1.18.1
  OsArch: linux/amd64
  Version: 3.4.4

• The OS

user1@test:~$ uname -a
Linux test 5.15.0-122-generic #132-Ubuntu SMP Thu Aug 29 13:45:52 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux

Seemed the debug log said the configuration of user2 did not take effect

DEBU[0000] Initializing boltdb state at /var/lib/sharedimages/libpod/bolt_state.db
DEBU[0000] Overriding run root "/run/user/1002" with "/run/user/1001/containers" from database
DEBU[0000] Overriding tmp dir "/run/user/1002/libpod/tmp" with "/run/user/1001/libpod/tmp" from databas

Then what's the expected configurations for this requirement?

Thank you very much!

Edit on 29th Sep.

I removed podman -> removed /var/lib/sharedimages -> removed /var/lib/containers -> removed /home/user1/.config/containers/storage.conf -> removed /home/user2/.config/container/stoarge.conf -> reboot host -> reinstalled podman -> recreated /var/lib/sharedimages and /var/lib/container

• try user2 first

user2@test:~$ podman images --log-level=debug
INFO[0000] podman filtering at log level debug
DEBU[0000] Called images.PersistentPreRunE(podman images --log-level=debug)
DEBU[0000] overlay storage already configured with a mount-program
DEBU[0000] Merged system config "/usr/share/containers/containers.conf"
DEBU[0000] overlay storage already configured with a mount-program
DEBU[0000] Using conmon: "/usr/bin/conmon"
DEBU[0000] Initializing boltdb state at /home/user2/.local/share/containers/storage/libpod/bolt_state.db
DEBU[0000] systemd-logind: Unknown object '/'.
DEBU[0000] Using graph driver overlay
DEBU[0000] Using graph root /home/user2/.local/share/containers/storage
DEBU[0000] Using run root /run/user/1002/containers
DEBU[0000] Using static dir /home/user2/.local/share/containers/storage/libpod
DEBU[0000] Using tmp dir /run/user/1002/libpod/tmp
DEBU[0000] Using volume path /home/user2/.local/share/containers/storage/volumes
DEBU[0000] overlay storage already configured with a mount-program
DEBU[0000] Set libpod namespace to ""
DEBU[0000] [graphdriver] trying provided driver "overlay"
DEBU[0000] overlay: mount_program=/usr/bin/fuse-overlayfs
DEBU[0000] backingFs=extfs, projectQuotaSupported=false, useNativeDiff=false, usingMetacopy=false
DEBU[0000] Initializing event backend journald
DEBU[0000] configured OCI runtime runc initialization failed: no valid executable found for OCI runtime runc: invalid argument
DEBU[0000] configured OCI runtime kata initialization failed: no valid executable found for OCI runtime kata: invalid argument
DEBU[0000] configured OCI runtime runsc initialization failed: no valid executable found for OCI runtime runsc: invalid argument
DEBU[0000] Using OCI runtime "/usr/bin/crun"
INFO[0000] Found CNI network podman (type=bridge) at /home/user2/.config/cni/net.d/87-podman.conflist
DEBU[0000] Default CNI network name podman is unchangeable
INFO[0000] Setting parallel job count to 37
REPOSITORY  TAG         IMAGE ID    CREATED     SIZE
DEBU[0000] Called images.PersistentPostRunE(podman images --log-level=debug)
user2@test:~$

Then user1

user1@test:~$ podman images --log-level=debug
INFO[0000] podman filtering at log level debug
DEBU[0000] Called images.PersistentPreRunE(podman images --log-level=debug)
DEBU[0000] Merged system config "/usr/share/containers/containers.conf"
DEBU[0000] Using conmon: "/usr/bin/conmon"
Error: error creating runtime static files directory: mkdir /var/lib/sharedimages/libpod: permission denied
user1@test:~$
user1@test:~$
user1@test:~$ more .config/c
cni/        containers/
user1@test:~$ more .config/containers/storage.conf
[storage]
driver = "overlay"
runroot = "/run/user/1001"
graphroot = "/var/lib/sharedimages"
[storage.options]
mount_program = "/bin/fuse-overlayfs"

[rhatdan]

@giuseppe PTAL