RKE2 and Kubernetes debug tips

Debug running containers

kubectl exec is obviously one way to see what happens in a running container, but if the image lacks tools for debugging, use ephemeral containers

Enable selinux with RKE2

Installing RKE2 with selinux enabled on SL Micro 6.2 requires some tricks:

• Add the rke2 folders to transactional-update mounts:

mkdir /etc/tukit.conf.d/
cat >/etc/tukit.conf.d/rancher.conf <<EOF
BINDDIRS[rancher]="/var/lib/rancher"
BINDDIRS[kubelet]="/var/lib/kubelet"
EOF
mkdir -p /var/lib/{rancher,kubelet}

• Get the rke2-selinux-VERSION.slemicro.noarch.rpm RPM package from the release in https://github.com/rancher/rke2-selinux/releases.
• run transactional-update pkg install rke2-selinux-XXX.rpm and reboot
• Configure rke2-server before installing it by adding selinux: true to /etc/rancher/rke2/config.yaml

The rest is common with installation without selinux:

• Install RKE2: curl -sfL https://get.rke2.io | INSTALL_RKE2_VERSION=v1.35.0+rke2r1 sh -
• Enable and start it: systemctl enable --now rke2-server

Apply local path provisioner policy

If using the local-path-provisioner some more tuning is needed See this upstream issue. Create a custom policy file: /root/localpathpolicy.te

module localpathpolicy 1.0;

require {
    type usr_t;
    type init_t;
    type container_t;
    type container_var_lib_t;
    
    class dir { search write add_name create remove_name rmdir setattr getattr };
    class file { create open write append read unlink setattr getattr };
}

#============= container_t ==============
allow container_t container_var_lib_t:file { create open write append read setattr getattr unlink };
allow container_t container_var_lib_t:dir { add_name create remove_name rmdir setattr write search };
allow container_t init_t:dir search;
allow container_t usr_t:dir { add_name create remove_name rmdir setattr getattr write };
allow container_t usr_t:file { create unlink write setattr getattr };
allow container_t init_t:file { read open };

And apply it (replace /opt/local-path-provisioner with the path used by the provisioner if different from the default after a tarball installation):

checkmodule -M -m -o /root/localpathpolicy.mod /root/localpathpolicy.te
semodule_package -o /root/localpathpolicy.pp -m /root/localpathpolicy.mod
semodule -i /root/localpathpolicy.pp
semanage fcontext -a -t container_file_t "/opt/local-path-provisioner(/.*)?"
restorecon -R -v /opt/local-path-provisioner

Add SELinux policy for the server

The server container runs systemd and thus requires a cgroup mount. This needs to be allowed with an SELinux policy. Note that this policy is not needed on a proxy.

Create a custom policy file: /root/systemdcontainerpolicy.te

module systemdcontainerpolicy 1.0;

require {
    type container_t;
    type cgroup_t; 
    
    class dir { search write add_name create remove_name rmdir setattr getattr };
    class file { create open write append read unlink setattr getattr watch };
    class filesystem getattr;
}

#============= container_t ==============
allow container_t cgroup_t:dir { add_name create remove_name rmdir setattr write search getattr };
allow container_t cgroup_t:file { create open write append read setattr getattr unlink watch };
allow container_t cgroup_t:filesystem getattr;

And apply it:

checkmodule -M -m -o /root/systemdcontainerpolicy.mod /root/systemdcontainerpolicy.te
semodule_package -o /root/systemdcontainerpolicy.pp -m /root/systemdcontainerpolicy.mod
semodule -i /root/systemdcontainerpolicy.pp

Apply SELinux policy to RKE2

If journalctl -xeu rke2-server shows this warning

SELinux is enabled for rke2 but process is not running in context 'container_runtime_t', rke2-selinux policy may need to be applied

run

restorecon /opt/rke2/bin/rke2
systemctl restart rke2-server

Installing RKE2 then rke2-selinux should prevents it, but it needs to be tested.

Debug what happens with Traefik

Add the following to the /var/lib/rancher/rke2/server/manifests/uyuni-traefik.yaml file:

apiVersion: helm.cattle.io/v1
kind: HelmChartConfig
metadata:
  name: rke2-traefik
  namespace: kube-system
spec:
  valuesContent: |-

    # Add these to get more verbose logs
    logs:
      general:
        level: DEBUG
      access:
        enabled: true

    # Add these to enable the dashboard
    additionalArguments:
      - "--api.dashboard=true"
      - "--api.insecure=true"
    ingressRoute:
      dashboard:
        enabled: true

Wait for traefik to be reloaded before continuing.

The dashboard is not exposed to the outside of the cluster. To get access to it, run this command: kubectl port-forward -n kube-system $(kubectl get pod -n kube-system -l app.kubernetes.io/name=rke2-traefik -o name) 8080:8080 and point your browser to http://localhost:8080/dashboard.

Traefik access and debug logs are available with kubectl logs -n kube-system $(kubectl get pod -n kube-system -l app.kubernetes.io/name=rke2-traefik -o name)

Reinstalling gotchas

Wait until all the server or proxy persistent volumes are gone after uninstalling the helm release. If running helm install again too fast would lead to some PVC never being created.