[rootless] [quadlet] [selinux]

[podhorsky-ksj]

Hi, I have issue with permissions with selinux on podman containers.
For now I have selinux in permissive mode and I can see in cockpit selinux messages like:

SELinux is preventing valkey-server from write access on the directory valkey.

I used rules generated by udica from podman inspect. Here is cil file:

(block valkey
    (blockinherit container)
    (blockinherit restricted_net_container)
    (allow process redis_port_t ( tcp_socket (  name_bind ))) 
    (allow process container_file_t ( dir ( add_name create getattr ioctl lock open read remove_name rmdir search setattr write ))) 
    (allow process container_file_t ( file ( append create getattr ioctl lock map open read rename setattr unlink write ))) 
    (allow process container_file_t ( fifo_file ( getattr read write append ioctl lock open ))) 
    (allow process container_file_t ( sock_file ( append getattr open read write ))) 
    (allow process user_home_t ( dir ( add_name create getattr ioctl lock open read remove_name rmdir search setattr write ))) 
    (allow process user_home_t ( file ( append create getattr ioctl lock map open read rename setattr unlink write ))) 
    (allow process user_home_t ( fifo_file ( getattr read write append ioctl lock open ))) 
    (allow process user_home_t ( sock_file ( append getattr open read write )))

And quadlet file:

[Unit]
Description=Immich
Requires=immich-network.service
After=immich-network.service
StartLimitIntervalSec=90
StartLimitBurst=2

[Container]
ContainerName=immich_valkey
AutoUpdate=registry
Image=docker.io/valkey/valkey
Volume=%h/valkey:/data:z
#HealthCmd=redis-cli ping || exit 1
Exec=valkey-server --save 60 1 --loglevel warning
UserNS=keep-id:uid=1000,gid=1000
Network=immich
IP=10.0.0.4
SecurityLabelType=valkey.process

[Service]
Restart=always
RestartSec=5

[Install]
WantedBy=default.target

I'm beginning with selinux. I checked, processes are running with valkey.process, files in directory (mounted via volume) in host and in container seems to have correct permission, which is in cil file.
But selinux still reports these errors in all containers and I don't know why.
Can you help me?

[rhatdan]

I don't see why you need any of these. This should all work with standard SELinux policy in container-selinux.

You should not need SecurityLabelType=valkey.process

[podhorsky-ksj]

Before this (udica and process file), selinux reported the same issue. I'm doing some tries and test how to fix it and I have seen in audit log (cockpit):
When booting, selinux block everything, these messages comes at least every minute. But when I restart some service (quadlet), in this case valkey, the messages stops coming from that exact service. I'm not sure how selinux is reporting it (like I said, I never worked with selinux before), if restart will stop putting these issues in audit log, but it is strange.

But after reboot, everything is back to the same issue reporting state.

[podhorsky-ksj]

I removed the modules created by udica and it is still the same.
It is strange. It should not deny the access:

ls -laZ

drwxr-xr-x. 1 immich immich system_u:object_r:container_file_t:s0:c168,c597    16 Jul 11 10:41 valkey

ps -efZ | grep valkey

unconfined_u:unconfined_r:container_runtime_t:s0 immich 3580 1389  0 10:31 ? 00:00:00 /usr/bin/conmon --api-version 1 -c 685fe806a179b9348cd643a7e133a91869eb60a0d3f0a7c15416ab5ab2a4a8d8 -u 685fe806a179b9348cd643a7e133a91869eb60a0d3f0a7c15416ab5ab2a4a8d8 -r /usr/bin/crun -b /home/immich/.local/share/containers/storage/overlay-containers/685fe806a179b9348cd643a7e133a91869eb60a0d3f0a7c15416ab5ab2a4a8d8/userdata -p /run/user/1003/containers/overlay-containers/685fe806a179b9348cd643a7e133a91869eb60a0d3f0a7c15416ab5ab2a4a8d8/userdata/pidfile -n immich_valkey --exit-dir /run/user/1003/libpod/tmp/exits --persist-dir /run/user/1003/libpod/tmp/persist/685fe806a179b9348cd643a7e133a91869eb60a0d3f0a7c15416ab5ab2a4a8d8 --full-attach -l journald --log-level warning --syslog --runtime-arg --log-format=json --runtime-arg --log --runtime-arg=/run/user/1003/containers/overlay-containers/685fe806a179b9348cd643a7e133a91869eb60a0d3f0a7c15416ab5ab2a4a8d8/userdata/oci-log --conmon-pidfile /run/user/1003/containers/overlay-containers/685fe806a179b9348cd643a7e133a91869eb60a0d3f0a7c15416ab5ab2a4a8d8/userdata/conmon.pid --exit-command /usr/bin/podman --exit-command-arg --root --exit-command-arg /home/immich/.local/share/containers/storage --exit-command-arg --runroot --exit-command-arg /run/user/1003/containers --exit-command-arg --log-level --exit-command-arg warning --exit-command-arg --cgroup-manager --exit-command-arg systemd --exit-command-arg --tmpdir --exit-command-arg /run/user/1003/libpod/tmp --exit-command-arg --network-config-dir --exit-command-arg  --exit-command-arg --network-backend --exit-command-arg netavark --exit-command-arg --volumepath --exit-command-arg /home/immich/.local/share/containers/storage/volumes --exit-command-arg --db-backend --exit-command-arg sqlite --exit-command-arg --transient-store=false --exit-command-arg --runtime --exit-command-arg crun --exit-command-arg --storage-driver --exit-command-arg overlay --exit-command-arg --events-backend --exit-command-arg journald --exit-command-arg container --exit-command-arg cleanup --exit-command-arg --rm --exit-command-arg 685fe806a179b9348cd643a7e133a91869eb60a0d3f0a7c15416ab5ab2a4a8d8
system_u:system_r:container_t:s0:c266,c463 immich 3632 3580  0 10:31 ?     00:00:02 valkey-server *:6379

SELinux is preventing valkey-server from write access on the file /data/temp-22.rdb.

[rhatdan]

The problem is the labels are differnt on the file content versus the process label.

system_u:object_r:container_file_t:s0:c168,c597
system_u:system_r:container_t:s0:c266,c463

The file content of valkey needs to be changed to match s0:c266,c463 or to be s0

The :z and :Z on the volume should be doing this.

[podhorsky-ksj]

I thought, the level (s0:c168,c597) is only for mls. I have default fedora type - targeted.
I have label Z on every volume every container. I tried to change it to z, but no difference.

if the level is taken for targeted too, does it have to do with UserNS=keep-id:uid=1000,gid=1000 ?

And if you mean the domain labels, articles on google said, it should be like this. that container_t has access to container_file_t. I haven't changed these labels manually.

[rhatdan]

The s0:c168,c597 is for MCS (Multi Category Security) and is used to keep containers separate.

Read about it in coloring book: https://people.redhat.com/duffy/selinux/selinux-coloring-book_A4-Stapled.pdf

Is the offending file in $HOME/data? Could you run restorecon -R -F $HOME/data and reset the labels on the entire directory then run the container again.

[podhorsky-ksj]

Yes, it is in home directory, see the quadlet file above:

Volume=%h/valkey:/data:z

you probably meant $HOME/valkey

I tried restorecon of $HOME/valkey, run container again and thenrebooted (to check new rows in audit log, which is always missing after restart of service) and issue is still the same.

[rhatdan]

what does
ls -Z $HOME/valkey

show?

[podhorsky-ksj]

Hmm, that is strange, i tried it again, just for check the level for process and file, for posting it here"
here is dir and process after apply restorecon:

system_u:system_r:container_t:s0:c17,c770 immich 3762 3741  0 15:55 ?      00:00:04 valkey-server *:6379
drwxr-xr-x. 1 immich immich unconfined_u:object_r:user_home_t:s0               16 Jul 11 16:14 valkey

after restart

drwxr-xr-x. 1 immich immich system_u:object_r:container_file_t:s0:c627,c915    16 Jul 11 16:15 valkey
system_u:system_r:container_t:s0:c627,c915 immich 10805 10802  0 16:15 ?   00:00:00 valkey-server *:6379

and after reboot:

system_u:system_r:container_t:s0:c627,c915 immich 10805 10802  0 16:15 ?   00:00:02 valkey-server *:6379
drwxr-xr-x. 1 immich immich system_u:object_r:container_file_t:s0:c627,c915    16 Jul 11 16:25 valkey

It seems to be stable now. I don't know how, but I will try to apply it the same way to other containers and will see.

[podhorsky-ksj]

Ok, so test is this:

restorecon -R -F /home/* && reboot

system_u:system_r:container_t:s0:c743,c892 immich 3366 3343  0 18:47 ?     00:00:00 valkey-server *:6379
drwxr-xr-x. 1 immich immich system_u:object_r:container_file_t:s0:c302,c408    16 Jul 11 18:49 valkey

restorecon -R -F /home/immich/valkey && reboot

drwxr-xr-x. 1 immich immich system_u:object_r:container_file_t:s0:c512,c966    16 Jul 11 18:55 valkey
system_u:system_r:container_t:s0:c269,c355 immich 3997 3972  0 18:54 ?     00:00:00 valkey-server *:6379

restorecon -R -F /home/immich/valkey && restart service

system_u:system_r:container_t:s0:c156,c381 immich 8198 8195  0 18:57 ?     00:00:00 valkey-server *:6379
drwxr-xr-x. 1 immich immich system_u:object_r:container_file_t:s0:c156,c381    16 Jul 11 18:57 valkey

restorecon -R -F /home/* && restart all services

drwxr-xr-x. 1 immich immich system_u:object_r:container_file_t:s0:c1,c397      16 Jul 11 19:02 valkey
system_u:system_r:container_t:s0:c1,c397 immich 9997 9994  0 19:02 ?       00:00:00 valkey-server *:6379

then reboot

system_u:system_r:container_t:s0:c207,c748 immich 3911 3826  0 19:04 ?     00:00:00 valkey-server *:6379
drwxr-xr-x. 1 immich immich system_u:object_r:container_file_t:s0:c197,c704    16 Jul 11 19:05 valkey

now when I know a little bit, that labeled should be only directories which is as volume into container, strange looking home, all files have the same label, even some of them have nothing to do with container. Directories library, postgres, valkey and cache are volumes to 4 different containers.

root@server-ksj:/home/immich# ls -laZ
total 60
drwx------. 1 immich immich system_u:object_r:container_file_t:s0:c197,c704   200 Jul 10 23:26 .
drwxr-xr-x. 1 root   root   system_u:object_r:home_root_t:s0                   72 Jul  4 13:02 ..
-rw-------. 1 immich immich system_u:object_r:container_file_t:s0:c197,c704  4094 Jul 11 18:58 .bash_history
-rw-r--r--. 1 immich immich system_u:object_r:container_file_t:s0:c197,c704    18 Feb  9 01:00 .bash_logout
-rw-r--r--. 1 immich immich system_u:object_r:container_file_t:s0:c197,c704   144 Feb  9 01:00 .bash_profile
-rw-r--r--. 1 immich immich system_u:object_r:container_file_t:s0:c197,c704   522 Feb  9 01:00 .bashrc
drwxr-xr-x. 1 immich immich system_u:object_r:container_file_t:s0:c197,c704     0 Jun 27 18:57 cache
drwxr-xr-x. 1 immich immich system_u:object_r:container_file_t:s0:c197,c704    34 Jul  4 13:11 .config
-rw-------. 1 immich immich system_u:object_r:container_file_t:s0:c197,c704    20 Jul  4 15:42 .lesshst
drwxr-xr-x. 1 immich immich system_u:object_r:container_file_t:s0:c197,c704    64 Jun 28 08:09 library
drwx------. 1 immich immich system_u:object_r:container_file_t:s0:c197,c704    10 Jul  4 15:38 .local
drwx------. 1 immich immich system_u:object_r:container_file_t:s0:c197,c704   570 Jul 11 19:04 postgres
drwxr-xr-x. 1 immich immich system_u:object_r:container_file_t:s0:c197,c704    16 Jul 11 19:05 valkey
-rw-------. 1 immich immich system_u:object_r:container_file_t:s0:c197,c704 16753 Jul  8 12:49 .viminfo

And once more restorecon -R -F /home/immich/valkey && restart service

drwxr-xr-x. 1 immich immich system_u:object_r:container_file_t:s0:c453,c918    16 Jul 11 19:13 valkey
system_u:system_r:container_t:s0:c453,c918 immich 8472 8469  0 19:13 ?     00:00:00 valkey-server *:6379

and then reboot - level labels were changed, but files are still compatible with process

system_u:system_r:container_t:s0:c361,c552 immich 3724 3683  0 19:15 ?     00:00:01 valkey-server *:6379
drwxr-xr-x. 1 immich immich system_u:object_r:container_file_t:s0:c361,c552    16 Jul 11 19:20 valkey

Result:
It seems everything except restorecon exact volume directory and restart service will result in bad level label, maybe not immediately, but after reboot for sure.

I'm sure, how now make containers working. But I'm not sure how to change home directory. I think everything shouldn't be labeled as container_file_t. I would expect user_home_t. But if I use restorecon in home directory, it will change it to next restart and then it will make containers bad.

Also, is this expected behavior? For me it seems pretty unstable. I cannot restart services by reboot, during the change context of selinux.

[rhatdan]

I was only looking for this once. If the content is being relabeled to container specific labels, then I would figure you are using :Z not :z, and this is only relabeling once?

The homedir being labeled incorrectly seems to show that you have forced a volume within a container and told it to relabel /home/immich badly.

Do you have ome/immich mounted into a container?

[podhorsky-ksj]

I'm using Z, in quadlet file above I used z, when I tested everything else.
I never mounted home/immich or other home directory in container.
Like I mentioned above, only library, postgres, valkey and cache from this directory are mounted to containers (4 different, each has only one volume).
I have done only what I mentioned in previous comment. Somehow home directory was relabeled differently after reboot.
I don't understand, how this relabeling works.

[rhatdan]

root@server-ksj:/home/immich# ls -laZ
total 60
drwx------. 1 immich immich system_u:object_r:container_file_t:s0:c197,c704   200 Jul 10 23:26 .
drwxr-xr-x. 1 root   root   system_u:object_r:home_root_t:s0                   72 Jul  4 13:02 ..
-rw-------. 1 immich immich system_u:object_r:container_file_t:s0:c197,c704  4094 Jul 11 18:58 .bash_history
-rw-r--r--. 1 immich immich system_u:object_r:container_file_t:s0:c197,c704    18 Feb  9 01:00 .bash_logout
-rw-r--r--. 1 immich immich system_u:object_r:container_file_t:s0:c197,c704   144 Feb  9 01:00 .bash_profile
-rw-r--r--. 1 immich immich system_u:object_r:container_file_t:s0:c197,c704   522 Feb  9 01:00 .bashrc
drwxr-xr-x. 1 immich immich system_u:object_r:container_file_t:s0:c197,c704     0 Jun 27 18:57 cache
drwxr-xr-x. 1 immich immich system_u:object_r:container_file_t:s0:c197,c704    34 Jul  4 13:11 .config
-rw-------. 1 immich immich system_u:object_r:container_file_t:s0:c197,c704    20 Jul  4 15:42 .lesshst
drwxr-xr-x. 1 immich immich system_u:object_r:container_file_t:s0:c197,c704    64 Jun 28 08:09 library
drwx------. 1 immich immich system_u:object_r:container_file_t:s0:c197,c704    10 Jul  4 15:38 .local

This is wrong and would only happen if this directory was volume mounted into a container with #22290
restoreon -R -v -F removes any changes to labels from the defaults. container_file_t is definitively not the default.

[podhorsky-ksj]

I haven't done it for sure. I want only minimum mounted to container. I want do have the most safest server.

As you can see from my comments above, restoreon -R -v -F remapped dir to

drwxr-xr-x. 1 immich immich unconfined_u:object_r:user_home_t:s0               16 Jul 11 16:14 valkey

but after reboot it was changed back to container_file_t.

Do you know how this relabeling works?

I'm still thinking if the directive

UserNS=keep-id:uid=1000,gid=1000 can cause this.

user in container is mapped to the same as in host. So, my host user immich, under who is everything running in container and under who is every directory under home/immich... Can it do something with it, when I used restorecon on home and it was looking for typical label?

One way or another, I would like some solution of this.
If I understand, restorecon will remove labels and then it looks for some after reboot. And when I remove quadlet files and container will not be started after reboot, it should not find the container label, but only home. And then I can return quadlet files and restart containers and see, what will happen with home. If it will be the same or everything changed back to container_file_t...

[rhatdan]

Well something is going very wrong if starting of a container is causing this mislabel.

restorecon the files back to user_home_t, and then restart the systemd service, to see if the files get chowned again.

[podhorsky-ksj]

After restorecon -R -F /home/immich/

root@server-ksj:/home/immich# ls -laZ
total 64
drwx------. 1 immich immich unconfined_u:object_r:user_home_dir_t:s0   200 Jul 10 23:26 .
drwxr-xr-x. 1 root   root   system_u:object_r:home_root_t:s0            72 Jul  4 13:02 ..
-rw-------. 1 immich immich unconfined_u:object_r:user_home_t:s0      4180 Jul 11 19:13 .bash_history
-rw-r--r--. 1 immich immich unconfined_u:object_r:user_home_t:s0        18 Feb  9 01:00 .bash_logout
-rw-r--r--. 1 immich immich unconfined_u:object_r:user_home_t:s0       144 Feb  9 01:00 .bash_profile
-rw-r--r--. 1 immich immich unconfined_u:object_r:user_home_t:s0       522 Feb  9 01:00 .bashrc
drwxr-xr-x. 1 immich immich unconfined_u:object_r:user_home_t:s0         0 Jun 27 18:57 cache
drwxr-xr-x. 1 immich immich unconfined_u:object_r:config_home_t:s0      34 Jul  4 13:11 .config
-rw-------. 1 immich immich unconfined_u:object_r:user_home_t:s0        20 Jul  4 15:42 .lesshst
drwxr-xr-x. 1 immich immich unconfined_u:object_r:user_home_t:s0        64 Jun 28 08:09 library
drwx------. 1 immich immich unconfined_u:object_r:gconf_home_t:s0       10 Jul  4 15:38 .local
drwx------. 1 immich immich unconfined_u:object_r:user_home_t:s0       570 Jul 11 19:15 postgres
drwxr-xr-x. 1 immich immich unconfined_u:object_r:user_home_t:s0        16 Jul 11 22:09 valkey
-rw-------. 1 immich immich unconfined_u:object_r:user_home_t:s0     16753 Jul  8 12:49 .viminfo

restart services

root@server-ksj:/home/immich# ls -laZ
total 64
drwx------. 1 immich immich unconfined_u:object_r:user_home_dir_t:s0           200 Jul 10 23:26 .
drwxr-xr-x. 1 root   root   system_u:object_r:home_root_t:s0                    72 Jul  4 13:02 ..
-rw-------. 1 immich immich unconfined_u:object_r:user_home_t:s0              4459 Jul 11 22:11 .bash_history
-rw-r--r--. 1 immich immich unconfined_u:object_r:user_home_t:s0                18 Feb  9 01:00 .bash_logout
-rw-r--r--. 1 immich immich unconfined_u:object_r:user_home_t:s0               144 Feb  9 01:00 .bash_profile
-rw-r--r--. 1 immich immich unconfined_u:object_r:user_home_t:s0               522 Feb  9 01:00 .bashrc
drwxr-xr-x. 1 immich immich system_u:object_r:container_file_t:s0:c923,c1012     0 Jun 27 18:57 cache
drwxr-xr-x. 1 immich immich unconfined_u:object_r:config_home_t:s0              34 Jul  4 13:11 .config
-rw-------. 1 immich immich unconfined_u:object_r:user_home_t:s0                20 Jul  4 15:42 .lesshst
drwxr-xr-x. 1 immich immich system_u:object_r:container_file_t:s0:c79,c199      64 Jun 28 08:09 library
drwx------. 1 immich immich unconfined_u:object_r:gconf_home_t:s0               10 Jul  4 15:38 .local
drwx------. 1 immich immich system_u:object_r:container_file_t:s0:c211,c801    570 Jul 11 22:11 postgres
drwxr-xr-x. 1 immich immich system_u:object_r:container_file_t:s0:c34,c597      16 Jul 11 22:10 valkey
-rw-------. 1 immich immich unconfined_u:object_r:user_home_t:s0             16753 Jul  8 12:49 .viminfo

reboot

root@server-ksj:/home/immich# ls -laZ
total 64
drwx------. 1 immich immich unconfined_u:object_r:user_home_dir_t:s0           200 Jul 10 23:26 .
drwxr-xr-x. 1 root   root   system_u:object_r:home_root_t:s0                    72 Jul  4 13:02 ..
-rw-------. 1 immich immich unconfined_u:object_r:user_home_t:s0              4459 Jul 11 22:11 .bash_history
-rw-r--r--. 1 immich immich unconfined_u:object_r:user_home_t:s0                18 Feb  9 01:00 .bash_logout
-rw-r--r--. 1 immich immich unconfined_u:object_r:user_home_t:s0               144 Feb  9 01:00 .bash_profile
-rw-r--r--. 1 immich immich unconfined_u:object_r:user_home_t:s0               522 Feb  9 01:00 .bashrc
drwxr-xr-x. 1 immich immich system_u:object_r:container_file_t:s0:c500,c681      0 Jun 27 18:57 cache
drwxr-xr-x. 1 immich immich unconfined_u:object_r:config_home_t:s0              34 Jul  4 13:11 .config
-rw-------. 1 immich immich unconfined_u:object_r:user_home_t:s0                20 Jul  4 15:42 .lesshst
drwxr-xr-x. 1 immich immich system_u:object_r:container_file_t:s0:c327,c1006    64 Jun 28 08:09 library
drwx------. 1 immich immich unconfined_u:object_r:gconf_home_t:s0               10 Jul  4 15:38 .local
drwx------. 1 immich immich system_u:object_r:container_file_t:s0:c551,c783    570 Jul 11 22:13 postgres
drwxr-xr-x. 1 immich immich system_u:object_r:container_file_t:s0:c381,c557     16 Jul 11 22:15 valkey
-rw-------. 1 immich immich unconfined_u:object_r:user_home_t:s0             16753 Jul  8 12:49 .viminfo

It seems level was relabeled after reboot, but other labels are still the same. Maybe it have to be relabeled every run of the container. At least home is ok.

hmm, I just remembered one thing. I really some homes put as volumes into container. Not for usual processes, but for backup. So, I have backup container which is doing this. I have it on timer, but because it is quadlet file, it cannot be disabled from running at startup. So, it can sometimes mix the labels, just when machine is rebooted.
Sorry, I didn't think about it earlier.

What should I do with it? I put there volumes as home directories, because I didn't want to put there a lot of directories to quadlet. It runs as user process under root. I'm not sure if it needs relabeling or I should use :z in every container i have.

[rhatdan]

Run this container with

SecurityLabelDisable=true

And do not use :z or :Z for volume mounting of $HOME

[podhorsky-ksj]

It seems to be working now, thanks. :-)
Btw, do you know how to give httpd_t access to websm_port_t? I want to have cockpit port under nginx. I can't find solution. I cannot modify the port (9090) under http_port_t, it will be blocked as:

type=AVC msg=audit(1720774813.308:48): avc:  denied  { name_bind } for  pid=1092 comm="(sd-listen)" src=9090 scontext=system_u:system_r:cockpit_ws_t:s0 tcontext=system_u:object_r:http_port_t:s0 tclass=tcp_socket permissive=1

[rhatdan]

Just add the allow rule to the cocpit_ws_t policy file.

[podhorsky-ksj]

I have found probably what you mean:

root@server-ksj:~# semodule -l | grep cockpit
cockpit
my-cockpitws

Which are probably two files my-cockpitws.pp and my-cockpitws.te in /root
One is binary, probably compiled the other one - my-cockpitws.te:

module my-cockpitws 1.0;

require {
        type container_file_t;
        type cockpit_ws_t;
        type default_t;
        class file { open read };
        class dir search;
}

#============= cockpit_ws_t ==============

#!!!! This avc is allowed in the current policy
allow cockpit_ws_t container_file_t:dir search;

#!!!! This avc is allowed in the current policy
allow cockpit_ws_t container_file_t:file { open read };

#!!!! This avc is allowed in the current policy
allow cockpit_ws_t default_t:file read;

So, the rule will be:

allow cockpit_ws_t httpd_t read;

?

And then build it with:

checkmodule -M –m –o my-cockpitws.mod my-cockpitws.te
semodule_package –m my-cockpitws.mod –o my-cockpitws.pp

and that's all?

[podhorsky-ksj]

Hmm, I have done it via:

ausearch -c 'nginx' --raw | grep 9090 | audit2allow -M my-nginx; semodule -i my-nginx.pp

without the grep in the middle compilation failed.
I will see if that works or not

[podhorsky-ksj]

It seems to be ok, Thanks a lot for help :-)