Rootless Podman NFS Storage (Nextcloud/MariaDB/Redis on Rocky 9 with Quadlet)

[Shadow8472]

I am rebuilding my homelab with Quadlets.
EDIT: Very important, I'm using a Synology NAS, which I believe isn't quite 1:1 with normal NFS implementations, having four squash options: [root|everyone] to [admin|guest]. At the moment of edit, I am using Everyone to Guest.

Working Quadlets:

• homelab-services.network
• caddy.container (Reverse proxy) [Domains from router]
• caddy-config.volume
• caddy-data.volume
• Vaultwarden.container (HTTPS from Caddy)
• vaultwarden-data.volume

Nextcloud Project (Notes in parentheses):

• nextcloud.pod.disabled (Podman v5.0.0 feature; Rocky 9 has v4.9.4)
• nextcloud.network
• nextcloud-fastvolume.volume (SSD storage)
• nextcloud-photovolume.volume (HDD storage)
• nextcloud-mariaDB.volume (NFS)
• nextcloud-nextcloud.container
• nextcloud-mariadb.container
• nextcloud-redis.container

Researched, but tentatively rejected:

• Ansible (Feels overweight for application)
• Podlet (Couldn't get to work quickly)

7/8 nextcloud* quadlets generate systemd --user unit files, except nextcloud.pod.
Nextcloud and MariaDB do not start.

$ systemctl --user start nextcloud-mariadb.service 
Job for nextcloud-mariadb.service failed because the control process exited with error code.
See "systemctl --user status nextcloud-mariadb.service" and "journalctl --user -xeu nextcloud-mariadb.service" for details.

$ systemctl --user status nextcloud-nextcloud.service
× nextcloud-nextcloud.service - Nextcloud Personal Cloud
     Loaded: loaded (/home/podmanuser/.config/containers/systemd/nextcloud/nextcloud-nextcloud.container; generated)
     Active: failed (Result: exit-code) since Wed 2024-10-02 01:37:08 PDT; 1min 36s ago
    Process: 102825 ExecStart=/usr/bin/podman run  [...] docker.io/library/nextcloud:latest (code=exited, status=125)
    Process: 102834 ExecStopPost=/usr/bin/podman rm -v -f -i --cidfile=/run/user/1001/nextcloud-nextcloud.cid (code=exited, status=0/SUCCESS)
   Main PID: 102825 (code=exited, status=125)
        CPU: 73ms

journalctl --user -xeu nextcloud-mariadb.service refuses to talk to underprivelaged users. Workaround: journalctl -xe in a privileged account.

Looks like I goofed with one error each from MariaDB and Nextcloud, but their counterparts are the same.

In a post from last year, I learned it was a permissions issue with NFS. #20519 After many suggestions, I was nudged toward Quadlet and using a file overlay when I got stuck trying to fix uid/gid numbers. I'm on Linux kernel version 5.14.0, which I read is supposed to have native file system overlay to help https://www.redhat.com/sysadmin/podman-rootless-overlay.

#nextcloud-nextcloud.container
[Unit]
Description=Nextcloud Personal Cloud
After=local-fs.target nextcloud-mariadb.service nextcloud-redis.service

[Container]
Label=app=nextcloud
ContainerName=NextCloud Home
#Pod=nextcloud.pod
Image=docker.io/library/nextcloud:latest
AutoUpdate=registry
Network=homelab-services.network
Network=nextcloud.network
Volume=nextcloud-fastvolume:/var/www/html
Volume=nextcloud-photovolume:/var/www/html/data/Photos
Environment=MARIADB_USER=nextclouddbuser
#Secret=password
#Environment=MARIADB_PASSWORD=/run/secrets/password
Environment=MARIADB_PASSWORD=mdbpass
#Secret=rootPassword
#Environment=MARIADB_ROOT_PASSWORD=/run/secrets/rootPassword
Environment=MARIADB_ROOT_PASSWORD=mdbrootpass

[Service]
Restart=always

[Install]
# Start by default on boot
WantedBy=default.target

#nextcloud.network
[Unit]
Description=NextCloud non-Pod Network
After=network-online.target

[Network]
Label=app=nextcloud
NetworkName=nextcloud
Subnet=10.98.75.0/24
Gateway=10.98.75.1

[Install]
WantedBy=default.target

Too long; didn't read?
I want rootless Nextcloud/MariaDB Podman containers doing their heavy weight data storage over NFS without.

Thanks!

[ygalblum]

Thanks for the detailed comment.

From what I can see, nextcloud-nextcloud.container does not have the NFS based volume. So, let's first focus on the MariaDB one. Can you share all its Quadlet files (i.e. .container and the rest of its dependencies)

[Shadow8472]

Synology support got back to me. Bad news up front, but at least a dead end is identified.

I understand you're looking into extended permissions, however we do not support NFS ACLs via NFS4. We typically suggest "Map all users to admin" for uniform permissions.

But, if you need non-uniform permissions you'd need to setup a kerberos server. This is to be able to map whichever users you may want to whatever user on DSM via ID mapping.

So, is Kerberos/LDAP worth looking into in my case?

[eriksjolund]

A part of mapping everyone to guest is because NFS doesn't understand namespaces.

My current understanding is that file systems do not need to understand user namespaces.
UIDs and GIDs are mapped before before the file system gets involved.
(I might be wrong)

But, if you need non-uniform permissions you'd need to setup a kerberos server.

Okay, the Synology NAS setting Map root to guest requires Kerberos/LDAP if understand the reply from Synology support correctly.

So, is Kerberos/LDAP worth looking into in my case?

Yes, I think rootless Podman will work better with Map root to guest than Map all users to guest
but I haven't done any tests to verify that as I don't have any Synology NAS.

I came to think about one test you could do even with Synology NAS setting Map all users to guest
You could try this syntax:

	     --uidmap +999:${intermediateUID}:1 \
	     --gidmap +999:${intermediateGID}:1 \

An example without NFS:

Click me

Here I use podman version 5.3.0-rc1

1. Run command to set shell variable

   intermediateUID=123
   

   The number was arbitrarily chosen
2. Run command to set shell variable

   intermediateGID=456
   

   The number was arbitrarily chosen
3. Run command to create a directory datadir
   with a specific ownership

   podman unshare bash -c \
    "mkdir datadir; chown ${intermediateUID}:${intermediateGID} datadir"
   

4. Check ownership of the directory

   podman unshare ls -ldn datadir
   

   The following output is printed

   drwxr-xr-x. 2 123 456 6 Oct 31 18:04 datadir
   

   result: the directory has ownership UID 123 and GID 456 when checked in in a user namespace with a standard mapping
5. Run mariadb container with the container UID 999 mapped to
   the intermediate UID ${intermediateUID} and container GID 999 mapped to
   the intermediate GID ${intermediateGID}

   podman run -d \
             --name mariadb \
         --env MARIADB_ALLOW_EMPTY_ROOT_PASSWORD=true \
         --uidmap +999:${intermediateUID}:1 \
         --gidmap +999:${intermediateGID}:1 \
         -v ./datadir:/var/lib/mysql:Z \
         docker.io/library/mariadb:latest
   

6. List files and directories in the directory datadir

   podman unshare ls -ln datadir
   

   The following output is printed

   total 162232
   -rw-rw----. 1 123 456   4890624 Oct 31 18:08 aria_log.00000001
   -rw-rw----. 1 123 456        52 Oct 31 18:08 aria_log_control
   -rw-rw----. 1 123 456         9 Oct 31 18:08 ddl_recovery.log
   -rw-rw----. 1 123 456       706 Oct 31 18:08 ib_buffer_pool
   -rw-rw----. 1 123 456 100663296 Oct 31 18:07 ib_logfile0
   -rw-rw----. 1 123 456  12582912 Oct 31 18:07 ibdata1
   -rw-rw----. 1 123 456  12582912 Oct 31 18:08 ibtmp1
   -rw-r--r--. 1 123 456        14 Oct 31 18:07 mariadb_upgrade_info
   -rw-rw----. 1 123 456         0 Oct 31 18:07 multi-master.info
   drwx------. 2 123 456      4096 Oct 31 18:08 mysql
   drwx------. 2 123 456        20 Oct 31 18:07 performance_schema
   drwx------. 2 123 456      8192 Oct 31 18:07 sys
   -rw-rw----. 1 123 456     24576 Oct 31 18:08 tc.log
   -rw-rw----. 1 123 456  10485760 Oct 31 18:07 undo001
   -rw-rw----. 1 123 456  10485760 Oct 31 18:07 undo002
   -rw-rw----. 1 123 456  10485760 Oct 31 18:07 undo003
   

7. List files and directories that are not owned by the UID ${intermediateUID}

   podman unshare find datadir -not -user ${intermediateUID}
   

   result: empty output. In other words, nothing was found
8. List files and directories that are not owned by the GID ${intermediateGID}

   podman unshare find datadir -not -group ${intermediateGID}
   

   result: empty output. In other words, nothing was found

So how could you make use of the syntax

	     --uidmap +999:${intermediateUID}:1 \
	     --gidmap +999:${intermediateGID}:1 \

with the Synology NAS setting Map all users to guest
?

I think the following instructions might work with the Synology NAS Map all users to guest setting, but I provide the instructions without guarantees as I don't have any Synology NAS to test with. Note, normally a user's subuids and subgids should be UIDs and GIDs only to be used for that user. Here we assign UID 1025 as subuid for the user test and GID 100 as subgid for the user test. This is suboptimal because other users on the client host have write access to 1025:100. The security consequences of this setup are a bit unclear to me.
Anyway, here the instructions:

Click me

1. Create user test

   sudo useradd test
   

2. Let UID 1025 be one of the subuids for the user test.

   sudo bash -c "echo $(id -un test):1025:1 >> /etc/subuid"
   

3. Let GID 100 be one of the subgids for the user test.

   sudo bash -c "echo $(id -un test):100:1 >> /etc/subgid"
   

4. Start a login shell for the user test

   sudo machinectl shell --uid=test
   

5. Create a directory in NFS

   mkdir /mnt/usbShare/datadir
   

6. Run command to set shell variable

   intermediateUID=$(podman unshare stat -c "%u" /mnt/usbShare/datadir)
   

7. Run command to set shell variable

   intermediateGID=$(podman unshare stat -c "%g" /mnt/usbShare/datadir)
   

8. Run mariadb container

   podman run -d \
             --name mariadb \
         --env MARIADB_ALLOW_EMPTY_ROOT_PASSWORD=true \
         --uidmap +999:${intermediateUID}:1 \
         --gidmap +999:${intermediateGID}:1 \
         --security-opt label=disable \
         -v /mnt/usbShare/datadir:/var/lib/mysql \
         docker.io/library/mariadb:latest
   

9. Check file user ownership

   find /mnt/usbShare/datadir -not -user 1025
   

   Hopefully nothing should be found
10. Check file group ownership

    find /mnt/usbShare/datadir -not -group 100
    

    Hopefully nothing should be found
11. Run command

    ls -l /mnt/usbShare/datadir
    

    Hopefully you will see some files and directories created by mariadb

[eriksjolund]

Regarding the error

 /entrypoint.sh: 111: cannot create /usr/local/etc/php/conf.d/redis-session.ini: Permission denied

I circumvented that error by creating an empty file ~/nextcloud-data/redis-session.ini and then bind-mounting it

Volume=%h/nextcloud-data/redis-session.ini:/usr/local/etc/php/conf.d/redis-session.ini:z

Reference:
https://github.com/eriksjolund/nextcloud-podman/blob/9b01bdfa282f84d6a5b4d18cd9df5d62ae9e0ea2/nextcloud.container#L22

In your setup probably remove :z

[Shadow8472]

Synology tech support got back and basically said they don't support extended permissions over NFS. ever. even with LDAP/Kerberos (If I understand him correctly). Try SMB if at all possible.

Truth be told, I'll need a day or two to sort through what either this or Synology official support said, but I did pick up an idea to try. I am going to try un-flattening permissions and making actual accounts for the different users MariaDB and Nextcloud... No, no Nextcloud's entrypoint.sh will still need the extended permissions. I think my best bet is to bind mount a modified copy of entrypoint.sh that doesn't care about extended permissions.

[Shadow8472]

I'm not ready to return to this project, but I found something that might be relevant.
https://github.com/derpibooru/philomena

If you have SELinux enforcing (Fedora, Arch, others; manifests as a Could not find a Mix.Project error), you should run the following in the application directory on the host before proceeding:

chcon -Rt svirt_sandbox_file_t .

This allows Docker or Podman to bind mount the application directory into the containers.
CHange CONtext.