Podman on Gentoo: system out of memory

[bennypowers]

Greetings! I've installed Gentoo (btw) on a Raspberry Pi 5, and configured the kernel according to both the Docker and Podman gentoo wiki articles. I then installed podman and set up a Quadlet service to run Nextcloud AIO. SELinux is not enabled.

When running the application, I might get one single successfuly page load of the app before it becomes unresponsive. At that point, the server typically shows ~100% memory usage and dozens of php-fpm processes.

In the journal logs for the nextcloud-aio-mastercontainer service, I noticed these lines:

conmon 6bd933d8ec6f64251dd6 <nwarn>: Failed to add inotify watch for /sys/fs/cgroup/user.slice/user-1000.slice/[email protected]/app.slice/nextcloud-aio-mastercontainer.service/libpod-payload-6bd933d8ec6f64251dd64c197af9e7d5599cb5875c31def857e6fc880b06ce72/memory.events
...
Trying to fix docker.sock permissions internally...
Adding internal www-data to group root
WARNING: No memory limit support
WARNING: No swap limit support
WARNING: No cpu cfs quota support
WARNING: No cpu cfs period support
WARNING: No cpu shares support
WARNING: No cpuset support
WARNING: IPv4 forwarding is disabled
Initial startup of Nextcloud All-in-One complete!

Full log

Feb 12 08:26:10 gentoo systemd[434]: Starting Nextcloud AIO Master Container...
Feb 12 08:26:13 gentoo nextcloud-aio-mastercontainer[558]: Trying to pull docker.io/nextcloud/all-in-one:latest...
Feb 12 08:26:13 gentoo nextcloud-aio-mastercontainer[558]: Pulling image //nextcloud/all-in-one:latest inside systemd: setting pull timeout to 5m0s
Feb 12 08:26:13 gentoo nextcloud-aio-mastercontainer[558]: Error: initializing source docker://nextcloud/all-in-one:latest: pinging container registry registry-1.docker.io: Get "https://registry-1.docker.io/v2/": dial tcp: lookup registry-1.docker.io: Temporary failure in name resolution
Feb 12 08:26:14 gentoo systemd[434]: nextcloud-aio-mastercontainer.service: Main process exited, code=exited, status=125/n/a
Feb 12 08:26:14 gentoo systemd[434]: nextcloud-aio-mastercontainer.service: Failed with result 'exit-code'.
Feb 12 08:26:14 gentoo systemd[434]: Failed to start Nextcloud AIO Master Container.
Feb 12 08:26:14 gentoo systemd[434]: nextcloud-aio-mastercontainer.service: Scheduled restart job, restart counter is at 1.
Feb 12 08:26:14 gentoo systemd[434]: Starting Nextcloud AIO Master Container...
Feb 12 08:26:15 gentoo nextcloud-aio-mastercontainer[649]: Trying to pull docker.io/nextcloud/all-in-one:latest...
Feb 12 08:26:15 gentoo nextcloud-aio-mastercontainer[649]: Pulling image //nextcloud/all-in-one:latest inside systemd: setting pull timeout to 5m0s
Feb 12 08:26:15 gentoo nextcloud-aio-mastercontainer[649]: Error: initializing source docker://nextcloud/all-in-one:latest: pinging container registry registry-1.docker.io: Get "https://registry-1.docker.io/v2/": dial tcp: lookup registry-1.docker.io: Temporary failure in name resolution
Feb 12 08:26:15 gentoo systemd[434]: nextcloud-aio-mastercontainer.service: Main process exited, code=exited, status=125/n/a
Feb 12 08:26:15 gentoo systemd[434]: nextcloud-aio-mastercontainer.service: Failed with result 'exit-code'.
Feb 12 08:26:15 gentoo systemd[434]: Failed to start Nextcloud AIO Master Container.
Feb 12 08:26:15 gentoo systemd[434]: nextcloud-aio-mastercontainer.service: Scheduled restart job, restart counter is at 2.
Feb 12 08:26:15 gentoo systemd[434]: Starting Nextcloud AIO Master Container...
Feb 12 08:26:15 gentoo nextcloud-aio-mastercontainer[759]: Trying to pull docker.io/nextcloud/all-in-one:latest...
Feb 12 08:26:15 gentoo nextcloud-aio-mastercontainer[759]: Pulling image //nextcloud/all-in-one:latest inside systemd: setting pull timeout to 5m0s
Feb 12 08:26:15 gentoo nextcloud-aio-mastercontainer[759]: Error: initializing source docker://nextcloud/all-in-one:latest: pinging container registry registry-1.docker.io: Get "https://registry-1.docker.io/v2/": dial tcp: lookup registry-1.docker.io: Temporary failure in name resolution
Feb 12 08:26:15 gentoo systemd[434]: nextcloud-aio-mastercontainer.service: Main process exited, code=exited, status=125/n/a
Feb 12 08:26:16 gentoo systemd[434]: nextcloud-aio-mastercontainer.service: Failed with result 'exit-code'.
Feb 12 08:26:16 gentoo systemd[434]: Failed to start Nextcloud AIO Master Container.
Feb 12 08:26:16 gentoo systemd[434]: nextcloud-aio-mastercontainer.service: Scheduled restart job, restart counter is at 3.
Feb 12 08:26:16 gentoo systemd[434]: Starting Nextcloud AIO Master Container...
Feb 12 08:26:16 gentoo nextcloud-aio-mastercontainer[859]: Trying to pull docker.io/nextcloud/all-in-one:latest...
Feb 12 08:26:16 gentoo nextcloud-aio-mastercontainer[859]: Pulling image //nextcloud/all-in-one:latest inside systemd: setting pull timeout to 5m0s
Feb 12 08:26:16 gentoo nextcloud-aio-mastercontainer[859]: Error: initializing source docker://nextcloud/all-in-one:latest: pinging container registry registry-1.docker.io: Get "https://registry-1.docker.io/v2/": dial tcp: lookup registry-1.docker.io: Temporary failure in name resolution
Feb 12 08:26:16 gentoo systemd[434]: nextcloud-aio-mastercontainer.service: Main process exited, code=exited, status=125/n/a
Feb 12 08:26:16 gentoo systemd[434]: nextcloud-aio-mastercontainer.service: Failed with result 'exit-code'.
Feb 12 08:26:16 gentoo systemd[434]: Failed to start Nextcloud AIO Master Container.
Feb 12 08:26:17 gentoo systemd[434]: nextcloud-aio-mastercontainer.service: Scheduled restart job, restart counter is at 4.
Feb 12 08:26:17 gentoo systemd[434]: Starting Nextcloud AIO Master Container...
Feb 12 08:26:17 gentoo nextcloud-aio-mastercontainer[970]: Trying to pull docker.io/nextcloud/all-in-one:latest...
Feb 12 08:26:17 gentoo nextcloud-aio-mastercontainer[970]: Pulling image //nextcloud/all-in-one:latest inside systemd: setting pull timeout to 5m0s
Feb 12 08:26:17 gentoo nextcloud-aio-mastercontainer[970]: Error: initializing source docker://nextcloud/all-in-one:latest: pinging container registry registry-1.docker.io: Get "https://registry-1.docker.io/v2/": dial tcp: lookup registry-1.docker.io: Temporary failure in name resolution
Feb 12 08:26:17 gentoo systemd[434]: nextcloud-aio-mastercontainer.service: Main process exited, code=exited, status=125/n/a
Feb 12 08:26:17 gentoo systemd[434]: nextcloud-aio-mastercontainer.service: Failed with result 'exit-code'.
Feb 12 08:26:17 gentoo systemd[434]: Failed to start Nextcloud AIO Master Container.
Feb 12 08:26:17 gentoo systemd[434]: nextcloud-aio-mastercontainer.service: Scheduled restart job, restart counter is at 5.
Feb 12 08:26:17 gentoo systemd[434]: nextcloud-aio-mastercontainer.service: Start request repeated too quickly.
Feb 12 08:26:17 gentoo systemd[434]: nextcloud-aio-mastercontainer.service: Failed with result 'exit-code'.
Feb 12 08:26:17 gentoo systemd[434]: Failed to start Nextcloud AIO Master Container.
Feb 12 08:47:14 gentoo systemd[434]: Starting Nextcloud AIO Master Container...
Feb 12 08:47:14 gentoo nextcloud-aio-mastercontainer[46296]: Trying to pull docker.io/nextcloud/all-in-one:latest...
Feb 12 08:47:14 gentoo nextcloud-aio-mastercontainer[46296]: Pulling image //nextcloud/all-in-one:latest inside systemd: setting pull timeout to 5m0s
Feb 12 08:47:17 gentoo nextcloud-aio-mastercontainer[46296]: Getting image source signatures
Feb 12 08:47:17 gentoo nextcloud-aio-mastercontainer[46296]: Copying blob sha256:8352cd48d6f5128d38ac5cbc0c30c63dec181d4b6e82e8729a06d0e4da6aac07
Feb 12 08:47:17 gentoo nextcloud-aio-mastercontainer[46296]: Copying blob sha256:8dbe855ad31e7cd79a1f706abb3a6b77f7e1457b5fb9b89c943ad0e8eb99d6a0
Feb 12 08:47:17 gentoo nextcloud-aio-mastercontainer[46296]: Copying blob sha256:668e1297a24eb8494667895e01ee28dd6cc44e981089460ea0e662c9462aab42
Feb 12 08:47:17 gentoo nextcloud-aio-mastercontainer[46296]: Copying blob sha256:dd4eb03fe552f5c8aa36cd79b78aa545592812821d1c3169cfeab7c2a2d2dfdc
Feb 12 08:47:17 gentoo nextcloud-aio-mastercontainer[46296]: Copying blob sha256:c6b39de5b33961661dc939b997cc1d30cda01e38005a6c6625fd9c7e748bab44
Feb 12 08:47:17 gentoo nextcloud-aio-mastercontainer[46296]: Copying blob sha256:1ecb7467a02f797b3b358a87cd793e73d849cb2dee4668faafaea6ac9037dc7b
Feb 12 08:47:17 gentoo nextcloud-aio-mastercontainer[46296]: Copying blob sha256:93b34edf664727029f0e28359adadc17c1e374090cbac579e5d6c3b120124063
Feb 12 08:47:17 gentoo nextcloud-aio-mastercontainer[46296]: Copying blob sha256:a410e0dcbdd51891d07a6f1e38b29551c9713d35cdfc5a03aedda38e5ecb4caa
Feb 12 08:47:17 gentoo nextcloud-aio-mastercontainer[46296]: Copying blob sha256:a849110782c82d3af560a8dd1a21dcf236ae3c7ae6eb9c712ac006bc07e5f276
Feb 12 08:47:17 gentoo nextcloud-aio-mastercontainer[46296]: Copying blob sha256:d2b560a4e1c6ec75a53d47e21254444d2f66c9be35d320411dadf7b78666c30d
Feb 12 08:47:17 gentoo nextcloud-aio-mastercontainer[46296]: Copying blob sha256:42931bdc49a45e00981c2e2429dd24a43fa54f475ffaabefc86ff37d198210c3
Feb 12 08:47:17 gentoo nextcloud-aio-mastercontainer[46296]: Copying blob sha256:1eb6ec6497cf80c1b6df930801d2cf6d2847e62575bfaae5970f615d458776f0
Feb 12 08:47:17 gentoo nextcloud-aio-mastercontainer[46296]: Copying blob sha256:461c9271732d41aadf8470db4a0c2f13170c856efce991ecce531f2d234994f5
Feb 12 08:47:17 gentoo nextcloud-aio-mastercontainer[46296]: Copying blob sha256:f39b6819f0058ca69b3578e5df8c8c6da607a61180607f97ec422a4c14295511
Feb 12 08:47:17 gentoo nextcloud-aio-mastercontainer[46296]: Copying blob sha256:070659c6d3616c651187e90accec619962df17c8cf5b1df46f5dbce991369fa0
Feb 12 08:47:17 gentoo nextcloud-aio-mastercontainer[46296]: Copying blob sha256:ef19b3bd31442e7732580ee1a2628b44b3ff6d90ec38d75b58b54d7ff63a342b
Feb 12 08:47:17 gentoo nextcloud-aio-mastercontainer[46296]: Copying blob sha256:60c628dafdb720383a61cac70d1c51680c2673a646cc80473d4617865e409bda
Feb 12 08:47:17 gentoo nextcloud-aio-mastercontainer[46296]: Copying blob sha256:d26d475282366dbb1f85e7f89526d07bee9d4a6f9aceb80ad6898dcc7d9760e7
Feb 12 08:47:17 gentoo nextcloud-aio-mastercontainer[46296]: Copying config sha256:de8ab1049901a74690aadc860970c6b175bc33dd97e76f96528b6357d68071c3
Feb 12 08:47:17 gentoo nextcloud-aio-mastercontainer[46296]: Writing manifest to image destination
Feb 12 08:47:17 gentoo podman[46296]: 2024-02-12 08:47:17.964321667 +0200 IST m=+3.634812091 image pull de8ab1049901a74690aadc860970c6b175bc33dd97e76f96528b6357d68071c3 docker.io/nextcloud/all-in-one:latest
Feb 12 08:47:18 gentoo podman[46296]: 2024-02-12 08:47:18.059944445 +0200 IST m=+3.730434813 container create 6bd933d8ec6f64251dd64c197af9e7d5599cb5875c31def857e6fc880b06ce72 (image=docker.io/nextcloud/all-in-one:latest, name=nextcloud-aio-mastercontainer, PODMAN_SYSTEMD_UNIT=nextcloud-aio-mastercontainer.service, io.containers.autoupdate=registry)
Feb 12 08:47:18 gentoo conmon[46400]: conmon 6bd933d8ec6f64251dd6 <nwarn>: Failed to add inotify watch for /sys/fs/cgroup/user.slice/user-1000.slice/[email protected]/app.slice/nextcloud-aio-mastercontainer.service/libpod-payload-6bd933d8ec6f64251dd64c197af9e7d5599cb5875c31def857e6fc880b06ce72/memory.events
Feb 12 08:47:18 gentoo podman[46296]: 2024-02-12 08:47:18.283398722 +0200 IST m=+3.953889109 container init 6bd933d8ec6f64251dd64c197af9e7d5599cb5875c31def857e6fc880b06ce72 (image=docker.io/nextcloud/all-in-one:latest, name=nextcloud-aio-mastercontainer, io.containers.autoupdate=registry, PODMAN_SYSTEMD_UNIT=nextcloud-aio-mastercontainer.service)
Feb 12 08:47:18 gentoo systemd[434]: Started Nextcloud AIO Master Container.
Feb 12 08:47:18 gentoo podman[46296]: 2024-02-12 08:47:18.322969056 +0200 IST m=+3.993459424 container start 6bd933d8ec6f64251dd64c197af9e7d5599cb5875c31def857e6fc880b06ce72 (image=docker.io/nextcloud/all-in-one:latest, name=nextcloud-aio-mastercontainer, io.containers.autoupdate=registry, PODMAN_SYSTEMD_UNIT=nextcloud-aio-mastercontainer.service)
Feb 12 08:47:18 gentoo nextcloud-aio-mastercontainer[46296]: 6bd933d8ec6f64251dd64c197af9e7d5599cb5875c31def857e6fc880b06ce72
Feb 12 08:47:18 gentoo nextcloud-aio-mastercontainer[46400]: Trying to fix docker.sock permissions internally...
Feb 12 08:47:18 gentoo nextcloud-aio-mastercontainer[46400]: Adding internal www-data to group root
Feb 12 08:47:22 gentoo nextcloud-aio-mastercontainer[46400]: WARNING: No memory limit support
Feb 12 08:47:22 gentoo nextcloud-aio-mastercontainer[46400]: WARNING: No swap limit support
Feb 12 08:47:22 gentoo nextcloud-aio-mastercontainer[46400]: WARNING: No cpu cfs quota support
Feb 12 08:47:22 gentoo nextcloud-aio-mastercontainer[46400]: WARNING: No cpu cfs period support
Feb 12 08:47:22 gentoo nextcloud-aio-mastercontainer[46400]: WARNING: No cpu shares support
Feb 12 08:47:22 gentoo nextcloud-aio-mastercontainer[46400]: WARNING: No cpuset support
Feb 12 08:47:22 gentoo nextcloud-aio-mastercontainer[46400]: WARNING: IPv4 forwarding is disabled
Feb 12 08:47:22 gentoo nextcloud-aio-mastercontainer[46400]: Initial startup of Nextcloud All-in-One complete!
Feb 12 08:47:22 gentoo nextcloud-aio-mastercontainer[46400]: You should be able to open the Nextcloud AIO Interface now on port 8080 of this server!
Feb 12 08:47:22 gentoo nextcloud-aio-mastercontainer[46400]: E.g. https://internal.ip.of.this.server:8080
Feb 12 08:47:22 gentoo nextcloud-aio-mastercontainer[46400]:
Feb 12 08:47:22 gentoo nextcloud-aio-mastercontainer[46400]: If your server has port 80 and 8443 open and you point a domain to your server, you can get a valid certificate automatically by opening the Nextcloud AIO Interface via:
Feb 12 08:47:22 gentoo nextcloud-aio-mastercontainer[46400]: https://your-domain-that-points-to-this-server.tld:8443
Feb 12 08:47:25 gentoo nextcloud-aio-mastercontainer[46400]: ++ head -1 /mnt/docker-aio-config/data/daily_backup_time
Feb 12 08:47:25 gentoo nextcloud-aio-mastercontainer[46400]: + BACKUP_TIME=04:00
Feb 12 08:47:25 gentoo nextcloud-aio-mastercontainer[46400]: + export BACKUP_TIME
Feb 12 08:47:25 gentoo nextcloud-aio-mastercontainer[46400]: + export DAILY_BACKUP=1
Feb 12 08:47:25 gentoo nextcloud-aio-mastercontainer[46400]: + DAILY_BACKUP=1
Feb 12 08:47:25 gentoo nextcloud-aio-mastercontainer[46400]: ++ sed -n 2p /mnt/docker-aio-config/data/daily_backup_time
Feb 12 08:47:25 gentoo nextcloud-aio-mastercontainer[46400]: + '[' '' '!=' automaticUpdatesAreNotEnabled ']'
Feb 12 08:47:25 gentoo nextcloud-aio-mastercontainer[46400]: + export AUTOMATIC_UPDATES=1
Feb 12 08:47:25 gentoo nextcloud-aio-mastercontainer[46400]: + AUTOMATIC_UPDATES=1
Feb 12 08:47:25 gentoo nextcloud-aio-mastercontainer[46400]: ++ sed -n 3p /mnt/docker-aio-config/data/daily_backup_time
Feb 12 08:47:25 gentoo nextcloud-aio-mastercontainer[46400]: + '[' '' '!=' successNotificationsAreNotEnabled ']'
Feb 12 08:47:25 gentoo nextcloud-aio-mastercontainer[46400]: + export SEND_SUCCESS_NOTIFICATIONS=1
Feb 12 08:47:25 gentoo nextcloud-aio-mastercontainer[46400]: + SEND_SUCCESS_NOTIFICATIONS=1
Feb 12 08:47:25 gentoo nextcloud-aio-mastercontainer[46400]: + set +x
Feb 12 08:47:25 gentoo nextcloud-aio-mastercontainer[46400]: {"level":"info","ts":1707720445.2967346,"msg":"using provided configuration","config_file":"/Caddyfile","config_adapter":""}
Feb 12 08:47:25 gentoo nextcloud-aio-mastercontainer[46400]: [12-Feb-2024 06:47:25] NOTICE: fpm is running, pid 132
Feb 12 08:47:25 gentoo nextcloud-aio-mastercontainer[46400]: [12-Feb-2024 06:47:25] NOTICE: ready to handle connections
Feb 12 08:47:25 gentoo nextcloud-aio-mastercontainer[46400]: [Mon Feb 12 06:47:25.631842 2024] [mpm_event:notice] [pid 119:tid 140736022175840] AH00489: Apache/2.4.58 (Unix) OpenSSL/3.1.4 configured -- resuming normal operations
Feb 12 08:47:25 gentoo nextcloud-aio-mastercontainer[46400]: [Mon Feb 12 06:47:25.631994 2024] [core:notice] [pid 119:tid 140736022175840] AH00094: Command line: 'httpd -D FOREGROUND'

In particular the inotify watch failure and the support warnings caught my eye. I suspect that something is not entirely kosher in my podman setup. I would be grateful to receive some guidance on how to proceed in debugging this.

These discussions may be related:

• Unable to check 'effective' cgroup limits #14538
• Is that possible to set resource limit in rootless mode when using cgroupfs? #19158