K3s Selinux with Cilium CNI Problem

[deiberts86]

PROBLEM: K3s Selinux doesn't seem to work well with Cilium CNI on K3s.

• OS: Rocky9 x86_64
• Kubernetes: K3s version 1.28.10

Installed container-selinux and k3s-selinux:

rpm -Ivh https://github.com/k3s-io/k3s-selinux/releases/download/v1.5.stable.1/k3s-selinux-1.5-1.el9.noarch.rpm

Current K3s config.yaml file:

cluster-init: true
disable-network-policy: true
disable-cloud-controller: true
disable-kube-proxy: true
flannel-backend: "none"
disable:
- traefik
- servicelb
node-ip: "$NODEIP"
node-external-ip: "$NODEEXTIP"
selinux: true
secrets-encryption: true
write-kubeconfig-mode: 0600
kube-controller-manager-arg:
- bind-address=127.0.0.1
- use-service-account-credentials=true
- tls-min-version=VersionTLS12
- tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384
kube-scheduler-arg:
- tls-min-version=VersionTLS12
- tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384
kube-apiserver-arg:
- tls-min-version=VersionTLS12
- tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384
- authorization-mode=RBAC,Node
- anonymous-auth=false
- admission-control-config-file=/etc/rancher/k3s/rancher-psact.yaml
- audit-policy-file=/etc/rancher/k3s/audit-policy.yaml
- audit-log-mode=blocking-strict
- audit-log-maxage=30
kubelet-arg:
- protect-kernel-defaults=true
- read-only-port=0
- authorization-mode=Webhook
- streaming-connection-idle-timeout=5m
token: "$TOKEN"
tls-san:
- $VIPSAN
- $VIP

• Installation of Cilium goes through just fine as expect but fails on SELinux with RUNC

audit2why -w -t AVC -i /var/log/audit/audit.log | less

• Output sample from Audit of SELinux

type=AVC msg=audit(1718396651.021:645): avc:  denied  { transition } for  pid=35709 comm="runc:[2:INIT]" path="/usr/bin/cilium-envoy-starter" dev="overlay" ino=125840900 scontext=system_u:system_r:unconfined_service_t:s0 tcontext=system_u:system_r:spc_t:s0 tclass=process permissive=0
type=AVC msg=audit(1718396652.028:663): avc:  denied  { transition } for  pid=35826 comm="runc:[2:INIT]" path="/usr/bin/cilium-envoy-starter" dev="overlay" ino=125840900 scontext=system_u:system_r:unconfined_service_t:s0 tcontext=system_u:system_r:spc_t:s0 tclass=process permissive=0
type=AVC msg=audit(1718396666.574:685): avc:  denied  { transition } for  pid=36045 comm="runc:[2:INIT]" path="/usr/bin/cilium-envoy-starter" dev="overlay" ino=125840900 scontext=system_u:system_r:unconfined_service_t:s0 tcontext=system_u:system_r:spc_t:s0 tclass=process permissive=0
type=AVC msg=audit(1718396693.570:715): avc:  denied  { transition } for  pid=36258 comm="runc:[2:INIT]" path="/usr/bin/cilium-envoy-starter" dev="overlay" ino=125840900 scontext=system_u:system_r:unconfined_service_t:s0 tcontext=system_u:system_r:spc_t:s0 tclass=process permissive=0
type=AVC msg=audit(1718396743.570:737): avc:  denied  { transition } for  pid=36387 comm="runc:[2:INIT]" path="/usr/bin/cilium-envoy-starter" dev="overlay" ino=125840900 scontext=system_u:system_r:unconfined_service_t:s0 tcontext=system_u:system_r:spc_t:s0 tclass=process permissive=0
type=AVC msg=audit(1718396827.573:759): avc:  denied  { transition } for  pid=36519 comm="runc:[2:INIT]" path="/usr/bin/cilium-envoy-starter" dev="overlay" ino=125840900 scontext=system_u:system_r:unconfined_service_t:s0 tcontext=system_u:system_r:spc_t:s0 tclass=process permissive=0
type=AVC msg=audit(1718396997.570:787): avc:  denied  { transition } for  pid=36713 comm="runc:[2:INIT]" path="/usr/bin/cilium-envoy-starter" dev="overlay" ino=125840900 scontext=system_u:system_r:unconfined_service_t:s0 tcontext=system_u:system_r:spc_t:s0 tclass=process permissive=0
type=AVC msg=audit(1718397304.559:815): avc:  denied  { transition } for  pid=36922 comm="runc:[2:INIT]" path="/usr/bin/cilium-envoy-starter" dev="overlay" ino=125840900 scontext=system_u:system_r:unconfined_service_t:s0 tcontext=system_u:system_r:spc_t:s0 tclass=process permissive=0

• When I set the overall SELinux policy to Permissive and restart server, it works like a champ. Can anyone look into this?

[dronenb]

@deiberts86 check the SELinux labels of the k3s binary:

ls -Z /usr/local/bin/k3s

Output should be similar to the following:

unconfined_u:object_r:container_runtime_exec_t:s0 /usr/local/bin/k3s

If not, try restoring the SELinux labels on that file using the following:

/usr/sbin/restorecon /usr/local/bin/k3s

A primer on SELinux that I found very helpful: https://www.youtube.com/watch?v=_WOKRaM-HI4

[deiberts86]

@dronenb, I will most certainly give that a try. I assumed it should have worked if installing the k3s-selinux rpm first then install k3s afterwards. I'll post the results soon when I have a free moment.