Some rootless containers stopped working

[timur-g]

First I appologize for low quality question "Rootless containers stopped working". But I am beginner user and no idea how to pinpoint.
Some months ago I created few rootless containers and they worked, also used named volumes so that I can upgrade image version.
OS is Linux Mint 21.3 (based on Ubnutu 22.04). I used built-in podman version 3.4, then upgraded in PPA to 6.x then today when this happened reverted to 3.4.4.

Today wanted to start the same containers, MariaDB runs but some do not start, notably Nextcloud and Collabora Online. Common is they have remapped ports >1024, if that is an indication. In later, coolwsd.xml is the main conf. file inside and without it container does not start.
For example (using old image I had I tried to create new container):
podman run --rm -it -p 9080:9980 -e "extra_params=--o:ssl.enable=false" --name 24.04.1-test collabora/code:24.04.1.4.1
gives:

 Cannot write random bytes:
404715AEB27F0000:error:12000079:random number generator:RAND_write_file:Cannot open file:../crypto/rand/randfile.c:240:Filename=/opt/cool/.rnd
Failed to initialize COOLWSD: Access to file denied: /etc/coolwsd/coolwsd.xml
-00001 2024-06-12 20:32:34.587063 +0000 [ coolwsd ] FTL  Failed to initialize COOLWSD: Access to file denied: /etc/coolwsd/coolwsd.xml| wsd/COOLWSD.hpp:535
Access to file denied: /etc/coolwsd/coolwsd.xml

I must have changed something in the meantime in the system, but cannot find what (user namespace?). This looks like a permission issue but where.
Help is more than welcome.

I looked issues here but cannot see if related, mostly they are above my level. Similar is #18300 but how to test that root is mapped properly..

cat /etc/subuid  
t:100000:65536
cat /etc/subgid
t:100000:65536

newfstatat(AT_FDCWD, "/usr/share/containers/containers.conf", {st_dev=makedev(0x103, 0x5), st_ino=11812191, st_mode=S_IFREG|0644, st_nlink=1, st_uid=65534, st_gid=65534, st_blksize=4096, st_blocks=40, st_size=17241, st_atime=1718220179 /* 2024-06-12T21:22:59.448670317+0200 */, st_atime_nsec=448670317, st_mtime=1640228028 /* 2021-12-23T03:53:48+0100 */, st_mtime_nsec=0, st_ctime=1718220155 /* 2024-06-12T21:22:35.900670685+0200 */, st_ctime_nsec=900670685}, 0) = 0
newfstatat(AT_FDCWD, "/etc/containers/containers.conf", {st_dev=makedev(0x103, 0x5), st_ino=13762870, st_mode=S_IFREG|0644, st_nlink=1, st_uid=65534, st_gid=65534, st_blksize=4096, st_blocks=8, st_size=931, st_atime=1718143479 /* 2024-06-12T00:04:39.731102827+0200 */, st_atime_nsec=731102827, st_mtime=1714208577 /* 2024-04-27T11:02:57.962682931+0200 */, st_mtime_nsec=962682931, st_ctime=1714208577 /* 2024-04-27T11:02:57.962682931+0200 */, st_ctime_nsec=962682931}, 0) = 0
newfstatat(AT_FDCWD, "/etc/containers/containers.conf.d", 0xc00003dd78, AT_SYMLINK_NOFOLLOW) = -1 ENOENT (No such file or directory)
newfstatat(AT_FDCWD, "/home/t/.config/cni/net.d", {st_dev=makedev(0x103, 0x5), st_ino=6474411, st_mode=S_IFDIR|0755, st_nlink=2, st_uid=0, st_gid=0, st_blksize=4096, st_blocks=8, st_size=4096, st_atime=1718209692 /* 2024-06-12T18:28:12.373118058+0200 */, st_atime_nsec=373118058, st_mtime=1714209214 /* 2024-04-27T11:13:34.838661482+0200 */, st_mtime_nsec=838661482, st_ctime=1714209214 /* 2024-04-27T11:13:34.838661482+0200 */, st_ctime_nsec=838661482}, 0) = 0
epoll_pwait(4, [], 128, 0, NULL, 0)     = 0
newfstatat(AT_FDCWD, "/run/user/1000/containers", {st_dev=makedev(0, 0x33), st_ino=92, st_mode=S_IFDIR|0700, st_nlink=6, st_uid=0, st_gid=0, st_blksize=4096, st_blocks=0, st_size=120, st_atime=1718220542 /* 2024-06-12T21:29:02.790592581+0200 */, st_atime_nsec=790592581, st_mtime=1718220548 /* 2024-06-12T21:29:08.182592668+0200 */, st_mtime_nsec=182592668, st_ctime=1718220548 /* 2024-06-12T21:29:08.182592668+0200 */, st_ctime_nsec=182592668}, 0) = 0
geteuid()                               = 0
openat(AT_FDCWD, "/home/t/.local/share/containers/storage/overlay/.has-mount-program", O_RDONLY|O_CLOEXEC) = 7
epoll_ctl(4, EPOLL_CTL_ADD, 7, {events=EPOLLIN|EPOLLOUT|EPOLLRDHUP|EPOLLET, data={u32=2294798184, u64=140159962568552}}) = -1 EPERM (Operation not permitted)

[legobeat]

Do you have a storage.conf? Output of podman info?

If you are using overlay, does this mirror your experience?

[timur-g]

locate shows /usr/share/containers/storage.conf but that does not exist (removed when I downgraded?). There exists /usr/share/containers/containers.conf.
podman infor showws: "configFile: /home/t/.config/containers/storage.conf" - but that file does not exist.

IDK if I am using overlay, seems not, as $STORAGE/overlay/ does not exist so I would say it is fuse-overlayfs. But where is that defined, if storage.conf does not exist? I guess fuse-overlayfs is the only option.

As if #22810, I tried the same Containerfile and it was OK:

STEP 1/4: FROM alpine:3.19
Resolved "alpine" as an alias (/etc/containers/registries.conf.d/shortnames.conf)
Trying to pull docker.io/library/alpine:3.19...
Getting image source signatures
Copying blob 4abcf2066143 done  
Copying config 05455a0888 done  
Writing manifest to image destination
Storing signatures
STEP 2/4: RUN mkdir -p /foo && chmod 777 /foo && chown 1234:1234 /foo && ls -la /foo
total 8
drwxrwxrwx    2 1234     1234          4096 Jun 13 17:54 .
dr-xr-xr-x    1 root     root          4096 Jun 13 17:54 ..
--> 62875fa2499
STEP 3/4: USER 1234
--> 362663035c6
STEP 4/4: RUN ls -la /foo && touch /foo/bar
total 8
drwxrwxrwx    2 1234     1234          4096 Jun 13 17:54 .
dr-xr-xr-x    1 root     root          4096 Jun 13 17:54 ..
COMMIT testperms
--> be206b4b44c
Successfully tagged localhost/testperms:latest
be206b4b44cdf19a01a3a4d603c4c2e23b086dba1aa19d3c62a2e7450cf229c6

~ % podman info

host:
  arch: amd64
  buildahVersion: 1.23.1
  cgroupControllers:
  - memory
  - pids
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: 'conmon: /usr/bin/conmon'
    path: /usr/bin/conmon
    version: 'conmon version 2.0.25, commit: unknown'
  cpus: 16
  distribution:
    codename: virginia
    distribution: linuxmint
    version: "21.3"
  eventLogger: journald
  hostname: lt.local
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
  kernel: 5.15.0-112-generic
  linkmode: dynamic
  logDriver: journald
  memFree: 16258732032
  memTotal: 29167665152
  ociRuntime:
    name: crun
    package: 'crun: /usr/bin/crun'
    path: /usr/bin/crun
    version: |-
      crun version 1.14.4
      commit: a220ca661ce078f2c37b38c92e66cf66c012d9c1
      rundir: /run/user/1000/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +YAJL
  os: linux
  remoteSocket:
    exists: true
    path: /run/user/1000/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: false
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: 'slirp4netns: /usr/bin/slirp4netns'
    version: |-
      slirp4netns version 1.0.1
      commit: 6a7b16babc95b6a3056b33fb45b74a6f62262dd4
      libslirp: 4.6.1
  swapFree: 2147479552
  swapTotal: 2147479552
  uptime: 1h 21m 10.68s (Approximately 0.04 days)
plugins:
  log:
  - k8s-file
  - none
  - journald
  network:
  - bridge
  - macvlan
  volume:
  - local
registries:
  '[docker.io](<http://docker.io/>)':
    Blocked: false
    Insecure: false
    Location: internal.registry.mirror/docker
    MirrorByDigestOnly: false
    Mirrors: null
    Prefix: '[docker.io](<http://docker.io/>)'
  '[quay.io](<http://quay.io/>)':
    Blocked: false
    Insecure: false
    Location: internal.registry.mirror/quay
    MirrorByDigestOnly: false
    Mirrors: null
    Prefix: '[quay.io](<http://quay.io/>)'
  '[registry.access.redhat.com](<http://registry.access.redhat.com/>)':
    Blocked: false
    Insecure: false
    Location: internal.registry.mirror/redhat
    MirrorByDigestOnly: false
    Mirrors: null
    Prefix: '[registry.access.redhat.com](<http://registry.access.redhat.com/>)'
  search:
  - docker.io
  - quay.io
  - registry.access.redhat.com
  - registry.fedoraproject.org
store:
  configFile: /home/t/.config/containers/storage.conf
  containerStore:
    number: 6
    paused: 0
    running: 0
    stopped: 6
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /home/t/.local/share/containers/storage
  graphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Using metacopy: "false"
  imageStore:
    number: 6
  runRoot: /run/user/1000/containers
  volumePath: /home/t/.local/share/containers/storage/volumes
version:
  APIVersion: 3.4.4
  Built: 0
  BuiltTime: Thu Jan  1 01:00:00 1970
  GitCommit: ""
  GoVersion: go1.18.1
  OsArch: linux/amd64
  Version: 3.4.4

In the previous strace, I saw "st_uid=0, st_gid=0 " is that indicative of an error?

[legobeat]

IDK if I am using overlay, seems not, as $STORAGE/overlay/ does not exist so I would say it is fuse-overlayfs. But where is that defined, if storage.conf does not exist?

From the output, it looks like you should have a /home/t/.config/containers/storage.conf where it sets it to overlay (which can still be using fuse-overlayfs) stored in /home/t/.local/share/containers/storage.

[timur-g]

That file does not exist:

ls /home/t/.config/containers/storage.conf
ls: cannot access '/home/t/.config/containers/storage.conf': No such file or directory

 ls /home/t/.local/share/containers/storage                                         
defaultNetworkBackend  mounts	 overlay	     overlay-images  secrets	   tmp		volumes
libpod		       networks  overlay-containers  overlay-layers  storage.lock  userns.lock

Is there a way to recreate it?
I assume that switch between overlay and fuse-overlayfs is not possible. If it is please direct.

[legobeat]

This is documented in man containers-storage.conf.

AIUI, those aren't different: You have the overlay driver using fuse-overlayfs the mount program. Which you can explicitly declare, for example:

# storage.conf
[storage]
driver = "overlay"

[storage.options.overlay]
mount_program = "/usr/bin/fuse-overlayfs"

[timur-g]

I created it, but still have the same error. I am grateful that you look at this, any idea how to get back those containers to work?

$ man containers-storage.conf 
No manual entry for containers-storage.conf

[legobeat]

Here's what I have so far:

1. Could you try the same Containerfile and commands as this comment and see if it reproduces for you? If so it seems like we're affected by the same issue.

2. As workaround, changing to slow driver = "vfs" instead might work.

[timur-g]

I tried that podman build before (as seen above) but without storage.conf, now I added it including force_mask = "shared". No error, OK.
I cannot dcoker run Collabora even with driver = "vfs". I think that cool user inside creates files. I did restart podman service.

[legobeat]

openat(AT_FDCWD, "/home/t/.local/share/containers/storage/overlay/.has-mount-program", O_RDONLY|O_CLOEXEC) = 7
epoll_ctl(4, EPOLL_CTL_ADD, 7, {events=EPOLLIN|EPOLLOUT|EPOLLRDHUP|EPOLLET, data={u32=2294798184, u64=140159962568552}}) = -1 EPERM (Operation not permitted)

Can you run your mount_program as the rootless user fine outside of podman?

[timur-g]

I guess that mount_program is /usr/bin/fuse-overlayfs. I can run it (same user), says fuse-overlayfs: no mountpoint specified
I did not know how to really mount something.

[timur-g]

I tried podman system migrate but it did not help.

[rhatdan]

You need to reset storage if you want to use native-overlay.
This file

/home/t/.local/share/containers/storage/overlay/.has-mount-program

Indicates at some point that you used a mount-program on this storage and podman will refuse to use the storage without a mount-program until this file is gone.

[timur-g]

Content of .has-mount-program file is "true". If I remove it, still the same error - and the file is recreated.
My goal is to get existing containers to work if possible, regardless which file system is used.
Did I understand well that the only way to get podman working is to delete everything and create again?
But what happened and how can it happen again? it is like losing data.

[rhatdan]

Then the containers were created using fuse-overlay, which means you need to install it as I understand it or reset the system and start from scratch.

[timur-g]

Yes, containers were created using fuse-overlay because I started with podman 3.4.
Later I upgraded to 6.x and still later got this error . Revert to 3.4 did not help. (kernels were changed along the way)
Are you suggesting I should install fuse-overlayfs? It is installed. if not, please explain what to do.

[rhatdan]

If it is installed, then I do not know what is going on. @giuseppe ideas?

[timur-g]

If there are no further ideas here, I plan to reset all podman conf ( I assume podman system reset ) .
So, I will not be able to test and this should be closed.