Run as non-root, OpenShift support

[kenmoini]

Problem Statement:
As a best practice, it's not advised to run containers as the root user: https://www.armosec.io/blog/kubernetes-security-best-practices/#limit-application-privileges

This provides a point of privilege escalation that can impact the wider cluster. As a solution to this, platforms such as OpenShift run containers as a random UID/GID unless specified.

When a container is built with the filesystem being owned to the root user, such as /opt/cribl then only the root user can read/write to those locations, so in order to have the container deploy to a Ready state you must run the container as root.

Solution Option(s):

• Ideally the containers have a specific user created with a UID/GID higher than 1000. This requires a rebuild of the Cribl container. Then in the podSecurityContext, you would specify the runAsUser and runAsGroup parameters to target that user.
• If the containers MUST be run as root, then provide the RoleBinding to allow the containers to run as Privileged containers on OpenShift:

---
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: crible-priv
  namespace: cribl-stream
subjects:
  - kind: ServiceAccount
    name: ls-wg-pci-logstream-workergroup
    namespace: cribl-stream
  - kind: ServiceAccount
    name: ls-wg-system-metrics-logstream-workergroup
    namespace: cribl-stream
  - kind: ServiceAccount
    name: default
    namespace: cribl-stream
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: 'system:openshift:scc:privileged'

I don't see the Cribl container source in the GitHub org, otherwise I'd try to provide a patch to this end. Is there any requirement for running as root that is not apparent to me? Thoughts on these options?

[bdalpe]

@kenmoini this PR might be of interest to you: #166 In fact, it was created specifically for dealing with a customer deploying the Leader on OpenShift.

The second item is that setting the CRIBL_VOLUME_DIR environment variable to a PVC that is writable by the UID, this removes any need for writing to /opt/cribl

NB: There's no reason for running any Cribl container as privileged.

[kenmoini]

@bdalpe Ok that is excellent and good to know - I've figured out a way to deploy to OpenShift with a namespaced user who is only admin of the cribl-stream namespace, without any additional RBAC. Namely just need to disable creating the RBAC and set the CRIBL_VOLUME_DIR env var:

helm install ls-leader cribl/logstream-leader \
  --set "config.groups={pcilogs,system-metrics}" \
  --set config.token="ABCDEF01-1234-5678-ABCD-ABCDEF012345" \
  --set config.adminPassword="adminPasswordHere" \
  --set config.license="<license key>" \ # or delete for demo
  -n cribl-stream

helm install ls-wg-pci cribl/logstream-workergroup \
  --set config.host="ls-leader-internal" \
  --set config.tag="pcilogs" \
  --set config.token="ABCDEF01-1234-5678-ABCD-ABCDEF012345" \
  --set rbac.create="false" \
  --set env.CRIBL_VOLUME_DIR=/tmp/cribl \
  --set service.annotations."metallb\.universe\.tf/address-pool=lab-pool" \
  -n cribl-stream

helm install ls-wg-system-metrics cribl/logstream-workergroup \
  --set config.host="ls-leader-internal" \
  --set config.tag="system-metrics" \
  --set config.token="ABCDEF01-1234-5678-ABCD-ABCDEF012345" \
  --set rbac.create="false" \
  --set env.CRIBL_VOLUME_DIR=/tmp/cribl \
  --set service.annotations."metallb\.universe\.tf/address-pool=lab-pool" \
  -n cribl-stream

So since it works with a random UID/GID like how OpenShift deploys by default, I think the best solution would just to be that we add some additional documentation around the value modifications needed to deploy to OpenShift - whatcha think?

[wa77z]

As of OpenShift 4.14 it seems pod security admission is in an enforced mode. This causes all pods to define their scc. Which permissions should the SCC have to run cribl as non-root? We also allowed the namespace to run from the GUID range defined in the pod, and allowed access to our storage class, however the pods still fail to start.

[bdalpe]

@wa77z do you have any logs or events from OpenShift showing the reason for the pod failing to start?

[wa77z]

@wa77z do you have any logs or events from OpenShift showing the reason for the pod failing to start?

Error creating: pods "cribl-edge-" is forbidden: unable to validate against any security context constraint