[Need some insight] two separated podman environments for one rootless linux user

[tzuwei93]

I'm recently using podman(4.6.1) on a rhel server (the podman info is at the bottom). I tried to implement two separated podman systems for one rootless linux user, i.e. separating graphroot, runroot, static dir, network config dir of Podman, and root of Runc for unrelated applications, and using some podmanfn system reset command (provided below) to safely stop all related containers. It is kind of the concept of having a portable config for different applications, which might being mentioned by @Meister1593 in #18375. The reason why I think this way, or want this feature, is because I'd like to have podman environments for different use cases for one linux user.

the test result is that the containers separated kind of survived from podman system reset (which does not manually set the configs mentioned above), but you can not login to them (the containers) again, i.e. kind of the customized user in container is lost in the user namespace, and also they could not survive from the health check command.

My question is that does I forget some settings or still need to have some other knowledge related to Podman in order to achieve my scenario ?

function podmanfn()
{
  tmpdir=/run/user/$(id -u)/airflow/libpod/tmp
  mkdir -p $tmpdir

  networkdir=/some/path/networkconfig_dir
  mkdir -p $networkdir
 
  podman --log-level debug --tmpdir $tmpdir \
         --network-config-dir ${networkdir} \
         --root /some/path/to/custom/graphroot \
         --runtime-flag root=/run/user/$(id -u)/airflow/runc \
         --runroot /run/user/$(id -u)/airflow/containers "$@"
}

---

podmanfn info

[some_linux_user@rhel_hostname some_appname-install-service]$ podmanfn info
INFO[0000] podman filtering at log level debug
DEBU[0000] Called info.PersistentPreRunE(podman --log-level debug --tmpdir /run/user/some_linux_uid/some_appname/libpod/tmp --network-config-dir /appool/some_party/some_appname-install-service/networkconfig_dir --root /appool/some_party/podman-storage/some_appname --runtime-flag root=/run/user/some_linux_uid/some_appname/runc --runroot /run/user/some_linux_uid/some_appname/containers info)
DEBU[0000] Using conmon: "/usr/bin/conmon"
DEBU[0000] Initializing boltdb state at /appool/some_party/podman-storage/some_appname/libpod/bolt_state.db
DEBU[0000] systemd-logind: Unknown object '/'.
DEBU[0000] Using graph driver overlay
DEBU[0000] Using graph root /appool/some_party/podman-storage/some_appname
DEBU[0000] Using run root /run/user/some_linux_uid/some_appname/containers
DEBU[0000] Using static dir /appool/some_party/podman-storage/some_appname/libpod
DEBU[0000] Using tmp dir /run/user/some_linux_uid/some_appname/libpod/tmp
DEBU[0000] Using volume path /appool/some_party/podman-storage/some_appname/volumes
DEBU[0000] Using transient store: false
DEBU[0000] [graphdriver] trying provided driver "overlay"
DEBU[0000] Cached value indicated that overlay is supported
DEBU[0000] Cached value indicated that overlay is supported
DEBU[0000] Cached value indicated that metacopy is not being used
DEBU[0000] Cached value indicated that native-diff is usable
DEBU[0000] backingFs=extfs, projectQuotaSupported=false, useNativeDiff=true, usingMetacopy=false
DEBU[0000] Initializing event backend file
DEBU[0000] Configured OCI runtime crun-wasm initialization failed: no valid executable found for OCI runtime crun-wasm: invalid argument
DEBU[0000] Configured OCI runtime runj initialization failed: no valid executable found for OCI runtime runj: invalid argument
DEBU[0000] Configured OCI runtime youki initialization failed: no valid executable found for OCI runtime youki: invalid argument
DEBU[0000] Configured OCI runtime ocijail initialization failed: no valid executable found for OCI runtime ocijail: invalid argument
DEBU[0000] Configured OCI runtime crun initialization failed: no valid executable found for OCI runtime crun: invalid argument
DEBU[0000] Configured OCI runtime kata initialization failed: no valid executable found for OCI runtime kata: invalid argument
DEBU[0000] Configured OCI runtime runsc initialization failed: no valid executable found for OCI runtime runsc: invalid argument
DEBU[0000] Configured OCI runtime krun initialization failed: no valid executable found for OCI runtime krun: invalid argument
DEBU[0000] Using OCI runtime "/usr/bin/runc"
INFO[0000] Setting parallel job count to 25
DEBU[0000] Loading registries configuration "/etc/containers/registries.conf"
DEBU[0000] Loading registries configuration "/etc/containers/registries.conf.d/000-shortnames.conf"
DEBU[0000] Loading registries configuration "/etc/containers/registries.conf.d/001-rhel-shortnames.conf"
DEBU[0000] Loading registries configuration "/etc/containers/registries.conf.d/002-rhel-shortnames-overrides.conf"
host:
  arch: amd64
  buildahVersion: 1.31.3
  cgroupControllers:
  - cpuset
  - cpu
  - io
  - memory
  - pids
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon-2.1.6-1.module+el8.8.0+18098+9b44df5f.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.6, commit: 8c4ab5a095127ecc96ef8a9c885e0e1b14aeb11b'
  cpuUtilization:
    idlePercent: 96.4
    systemPercent: 1.82
    userPercent: 1.78
  cpus: 8
  databaseBackend: boltdb
  distribution:
    distribution: '"rhel"'
    version: "8.6"
  eventLogger: file
  freeLocks: 2038
  hostname: darhhdlen1
  idMappings:
    gidmap:
    - container_id: 0
      host_id: some_linux_gid
      size: 1
    - container_id: 1
      host_id: 2918048
      size: 65536
    uidmap:
    - container_id: 0
      host_id: some_linux_uid
      size: 1
    - container_id: 1
      host_id: 2918048
      size: 65536
  kernel: 4.18.0-372.32.1.el8_6.x86_64
  linkmode: dynamic
  logDriver: k8s-file
  memFree: 12273070080
  memTotal: 33479532544
  networkBackend: cni
  networkBackendInfo:
    backend: cni
    dns:
      package: podman-plugins-4.6.1-4.module+el8.9.0+20326+387084d0.x86_64
      path: /usr/libexec/cni/dnsname
      version: |-
        CNI dnsname plugin
        version: 1.3.1
        commit: unknown
    package: containernetworking-plugins-1.2.0-1.module+el8.8.0+18060+3f21f2cc.x86_64
    path: /usr/libexec/cni
  ociRuntime:
    name: runc
    package: runc-1.1.4-1.module+el8.8.0+18060+3f21f2cc.x86_64
    path: /usr/bin/runc
    version: |-
      runc version 1.1.4
      spec: 1.0.2-dev
      go: go1.19.4
      libseccomp: 2.5.2
  os: linux
  pasta:
    executable: ""
    package: ""
    version: ""
  remoteSocket:
    exists: true
    path: /run/user/some_linux_uid/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_SYS_CHROOT,CAP_NET_RAW,CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: false
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.2.0-2.module+el8.8.0+18060+3f21f2cc.x86_64
    version: |-
      slirp4netns version 1.2.0
      commit: 656041d45cfca7a4176f6b7eed9e4fe6c11e8383
      libslirp: 4.4.0
      SLIRP_CONFIG_VERSION_MAX: 3
      libseccomp: 2.5.2
  swapFree: 16709214208
  swapTotal: 16911429632
  uptime: 217h 60m 58.00s (Approximately 9.04 days)
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries:
  search:
  - registry.access.redhat.com
  - registry.redhat.io
  - docker.io
store:
  configFile: /home/some_linux_user/.config/containers/storage.conf
  containerStore:
    number: 8
    paused: 0
    running: 8
    stopped: 0
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /appool/some_party/podman-storage/some_appname
  graphRootAllocated: 105147981824
  graphRootUsed: 81086812160
  graphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 3
  runRoot: /run/user/some_linux_uid/some_appname/containers
  transientStore: false
  volumePath: /appool/some_party/podman-storage/some_appname/volumes
version:
  APIVersion: 4.6.1
  Built: 1696868155
  BuiltTime: Tue Oct 10 00:15:55 2023
  GitCommit: ""
  GoVersion: go1.20.6
  Os: linux
  OsArch: linux/amd64
  Version: 4.6.1

DEBU[0000] Called info.PersistentPostRunE(podman --log-level debug --tmpdir /run/user/some_linux_uid/some_appname/libpod/tmp --network-config-dir /appool/some_party/some_appname-install-service/networkconfig_dir --root /appool/some_party/podman-storage/some_appname --runtime-flag root=/run/user/some_linux_uid/some_appname/runc --runroot /run/user/some_linux_uid/some_appname/containers info)
DEBU[0000] Shutting down engines

podman info --log-level debug

[some_linux_user@rhel_hostname some_appname-install-service]$ podman info --log-level debug
INFO[0000] podman filtering at log level debug
DEBU[0000] Called info.PersistentPreRunE(podman info --log-level debug)
DEBU[0000] Using conmon: "/usr/bin/conmon"
DEBU[0000] Initializing boltdb state at /appool/cub/some_linux_uid/.local/share/containers/storage/libpod/bolt_state.db
DEBU[0000] Overriding graph root "/appool/cub/some_linux_uid/.local/share/containers/storage" with "/home/some_linux_uid/.local/share/containers/storage" from database
DEBU[0000] Overriding static dir "/appool/cub/some_linux_uid/.local/share/containers/storage/libpod" with "/home/some_linux_uid/.local/share/containers/storage/libpod" from database
DEBU[0000] Overriding volume path "/appool/cub/some_linux_uid/.local/share/containers/storage/volumes" with "/home/some_linux_uid/.local/share/containers/storage/volumes" from database
DEBU[0000] systemd-logind: Unknown object '/'.
DEBU[0000] Using graph driver overlay
DEBU[0000] Using graph root /home/some_linux_uid/.local/share/containers/storage
DEBU[0000] Using run root /run/user/some_linux_uid/containers
DEBU[0000] Using static dir /home/some_linux_uid/.local/share/containers/storage/libpod
DEBU[0000] Using tmp dir /run/user/some_linux_uid/libpod/tmp
DEBU[0000] Using volume path /home/some_linux_uid/.local/share/containers/storage/volumes
DEBU[0000] Using transient store: false
DEBU[0000] [graphdriver] trying provided driver "overlay"
DEBU[0000] Cached value indicated that overlay is supported
DEBU[0000] Cached value indicated that overlay is supported
DEBU[0000] Cached value indicated that metacopy is not being used
DEBU[0000] Cached value indicated that native-diff is usable
DEBU[0000] backingFs=extfs, projectQuotaSupported=false, useNativeDiff=true, usingMetacopy=false
DEBU[0000] Initializing event backend file
DEBU[0000] Configured OCI runtime crun initialization failed: no valid executable found for OCI runtime crun: invalid argument
DEBU[0000] Configured OCI runtime crun-wasm initialization failed: no valid executable found for OCI runtime crun-wasm: invalid argument
DEBU[0000] Configured OCI runtime kata initialization failed: no valid executable found for OCI runtime kata: invalid argument
DEBU[0000] Configured OCI runtime ocijail initialization failed: no valid executable found for OCI runtime ocijail: invalid argument
DEBU[0000] Configured OCI runtime runj initialization failed: no valid executable found for OCI runtime runj: invalid argument
DEBU[0000] Configured OCI runtime runsc initialization failed: no valid executable found for OCI runtime runsc: invalid argument
DEBU[0000] Configured OCI runtime youki initialization failed: no valid executable found for OCI runtime youki: invalid argument
DEBU[0000] Configured OCI runtime krun initialization failed: no valid executable found for OCI runtime krun: invalid argument
DEBU[0000] Using OCI runtime "/usr/bin/runc"
INFO[0000] Setting parallel job count to 25
DEBU[0000] Loading registries configuration "/etc/containers/registries.conf"
DEBU[0000] Loading registries configuration "/etc/containers/registries.conf.d/000-shortnames.conf"
DEBU[0000] Loading registries configuration "/etc/containers/registries.conf.d/001-rhel-shortnames.conf"
DEBU[0000] Loading registries configuration "/etc/containers/registries.conf.d/002-rhel-shortnames-overrides.conf"
host:
  arch: amd64
  buildahVersion: 1.31.3
  cgroupControllers:
  - cpuset
  - cpu
  - io
  - memory
  - pids
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon-2.1.6-1.module+el8.8.0+18098+9b44df5f.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.6, commit: 8c4ab5a095127ecc96ef8a9c885e0e1b14aeb11b'
  cpuUtilization:
    idlePercent: 96.4
    systemPercent: 1.82
    userPercent: 1.78
  cpus: 8
  databaseBackend: boltdb
  distribution:
    distribution: '"rhel"'
    version: "8.6"
  eventLogger: file
  freeLocks: 2038
  hostname: darhhdlen1
  idMappings:
    gidmap:
    - container_id: 0
      host_id: some_linux_gid
      size: 1
    - container_id: 1
      host_id: 2918048
      size: 65536
    uidmap:
    - container_id: 0
      host_id: some_linux_uid
      size: 1
    - container_id: 1
      host_id: 2918048
      size: 65536
  kernel: 4.18.0-372.32.1.el8_6.x86_64
  linkmode: dynamic
  logDriver: k8s-file
  memFree: 12260167680
  memTotal: 33479532544
  networkBackend: cni
  networkBackendInfo:
    backend: cni
    dns:
      package: podman-plugins-4.6.1-4.module+el8.9.0+20326+387084d0.x86_64
      path: /usr/libexec/cni/dnsname
      version: |-
        CNI dnsname plugin
        version: 1.3.1
        commit: unknown
    package: containernetworking-plugins-1.2.0-1.module+el8.8.0+18060+3f21f2cc.x86_64
    path: /usr/libexec/cni
  ociRuntime:
    name: runc
    package: runc-1.1.4-1.module+el8.8.0+18060+3f21f2cc.x86_64
    path: /usr/bin/runc
    version: |-
      runc version 1.1.4
      spec: 1.0.2-dev
      go: go1.19.4
      libseccomp: 2.5.2
  os: linux
  pasta:
    executable: ""
    package: ""
    version: ""
  remoteSocket:
    exists: true
    path: /run/user/some_linux_uid/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_SYS_CHROOT,CAP_NET_RAW,CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: false
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.2.0-2.module+el8.8.0+18060+3f21f2cc.x86_64
    version: |-
      slirp4netns version 1.2.0
      commit: 656041d45cfca7a4176f6b7eed9e4fe6c11e8383
      libslirp: 4.4.0
      SLIRP_CONFIG_VERSION_MAX: 3
      libseccomp: 2.5.2
  swapFree: 16709214208
  swapTotal: 16911429632
  uptime: 218h 3m 40.00s (Approximately 9.08 days)
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries:
  search:
  - registry.access.redhat.com
  - registry.redhat.io
  - docker.io
store:
  configFile: /home/some_linux_uid/.config/containers/storage.conf
  containerStore:
    number: 0
    paused: 0
    running: 0
    stopped: 0
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /home/some_linux_uid/.local/share/containers/storage
  graphRootAllocated: 105147981824
  graphRootUsed: 81086947328
  graphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 0
  runRoot: /run/user/some_linux_uid/containers
  transientStore: false
  volumePath: /home/some_linux_uid/.local/share/containers/storage/volumes
version:
  APIVersion: 4.6.1
  Built: 1696868155
  BuiltTime: Tue Oct 10 00:15:55 2023
  GitCommit: ""
  GoVersion: go1.20.6
  Os: linux
  OsArch: linux/amd64
  Version: 4.6.1

DEBU[0000] Called info.PersistentPostRunE(podman info --log-level debug)
DEBU[0000] Shutting down engines

[Luap99]

There is only one user namespace (pause process) per user. When you call podman system reset it will kill the pause process thus make other namespace incorrect.

You can do that stuff to isolate instances but don't use system reset or system migrate as they kill the pause process. If you do podman rm -all; podman image rm --all; podman volume rm --all etc... then it should still work.

[tzuwei93]

Got it, thanks for providing this info. :)