Issue with replicating certain K8S resources into OPA through Kube-mgmt

[aleksandar-ruskov]

Hi everyone, I have encountered an issue regarding replicating k8s resources into OPA's data through Kube-mgmt.

Previously I needed to iterate through the existing pods as well as deployments in my k8s cluster. So I added those resources for replication in the kube-mgmt configuration of the admission controller.

Even though adding the required args in kube-mgmt for pods and deployments worked, I faced another problem with other resources such as Roles, Secrets, etc. I followed the kube-mgmt readme when I was adding the K8S resources to be replicated in the admission controller (https://github.com/open-policy-agent/kube-mgmt/tree/0753a3f46f95a22062caf31e2c69b89d298e736e#readme).

Since Roles and Secrets did not seem to be replicated successfully and I couldn't access them through data.kubernetes.secrets or data.kubernetes.roles, I made an experiment where I try to replicate all K8S api resources and added all to the admission controller:

- name: kube-mgmt
          image: openpolicyagent/kube-mgmt:2.0.1
          args:
            - "--replicate-cluster=v1/componentstatuses"
            - "--replicate-cluster=v1/namespaces"
            - "--replicate-cluster=v1/nodes"
            - "--replicate-cluster=v1/persistentvolumes"
            - "--replicate-cluster=apiregistration.k8s.io/v1/apiservices"
            - "--replicate-cluster=authentication.k8s.io/v1/tokenreviews"
            - "--replicate-cluster=authorization.k8s.io/v1/selfsubjectaccessreviews"
            - "--replicate-cluster=authorization.k8s.io/v1/selfsubjectrulesreviews"
            - "--replicate-cluster=authorization.k8s.io/v1/subjectaccessreviews"
            - "--replicate-cluster=certificates.k8s.io/v1/certificatesigningrequests"
            - "--replicate-cluster=flowcontrol.apiserver.k8s.io/v1beta2/flowschemas"
            - "--replicate-cluster=flowcontrol.apiserver.k8s.io/v1beta2/prioritylevelconfigurations"
            - "--replicate-cluster=networking.k8s.io/v1/ingressclasses"
            - "--replicate-cluster=node.k8s.io/v1/runtimeclasses"
            - "--replicate-cluster=rbac.authorization.k8s.io/v1/clusterrolebindings"
            - "--replicate-cluster=rbac.authorization.k8s.io/v1/clusterroles"
            - "--replicate-cluster=scheduling.k8s.io/v1/priorityclasses"
            - "--replicate-cluster=storage.k8s.io/v1/csidrivers"
            - "--replicate-cluster=storage.k8s.io/v1/csinodes"
            - "--replicate-cluster=storage.k8s.io/v1/storageclasses"
            - "--replicate-cluster=storage.k8s.io/v1/volumeattachments"
            - "--replicate=v1/bindings"
            - "--replicate=v1/configmaps"
            - "--replicate=v1/endpoints"
            - "--replicate=v1/events"
            - "--replicate=v1/limitranges"
            - "--replicate=v1/persistentvolumeclaims"
            - "--replicate=v1/pods"
            - "--replicate=v1/podtemplates"
            - "--replicate=v1/replicationcontrollers"
            - "--replicate=v1/resourcequotas"
            - "--replicate=v1/secrets"
            - "--replicate=v1/serviceaccounts"
            - "--replicate=v1/services"
            - "--replicate=apps/v1/controllerrevisions"
            - "--replicate=apps/v1/daemonsets"
            - "--replicate=apps/v1/deployments"
            - "--replicate=apps/v1/replicasets"
            - "--replicate=apps/v1/statefulsets"
            - "--replicate=authorization.k8s.io/v1/localsubjectaccessreviews"
            - "--replicate=autoscaling/v2/horizontalpodautoscalers"
            - "--replicate=batch/v1/cronjobs"
            - "--replicate=batch/v1/jobs"
            - "--replicate=coordination.k8s.io/v1/leases"
            - "--replicate=discovery.k8s.io/v1/endpointslices"
            - "--replicate=events.k8s.io/v1/events"
            - "--replicate=networking.k8s.io/v1/ingresses"
            - "--replicate=networking.k8s.io/v1/networkpolicies"
            - "--replicate=policy/v1/poddisruptionbudgets"
            - "--replicate=rbac.authorization.k8s.io/v1/rolebindings"
            - "--replicate=rbac.authorization.k8s.io/v1/roles"
            - "--replicate=storage.k8s.io/v1/csistoragecapacities"

Then I tried to access each of them in a policy, where for each resource I follow the pattern:

  import data.kubernetes.<resource>
  deny[message]{
      not <resource>
      message := "<resource> is not correctly imported!"
  }

As a result the following resources triggered the policy (these seem to fail to import):

Cluster-level resources (overall only namespaces seem to be imported correctly):

• apiservices
• certificatesigningrequests
• clusterrolebindings
• clusterroles
• componentstatuses
• csidrivers
• csinodes
• flowschemas
• ingressclasses
• localsubjectaccessreviews
• nodes
• persistentvolumes
• priorityclasses
• prioritylevelconfigurations
• runtimeclasses
• selfsubjectaccessreviews
• selfsubjectrulesreviews
• storageclasses
• subjectaccessreviews
• token reviews
• volumeattachments

Namespace-level resources:

• bindings
• csistoragecapacities
• leases
• localsubjectaccessreviews
• podtemplates
• rolebindings
• roles
• secrets

So it is visible that most namespaces resources get imported although there are some which fail, but also almost all cluster-level resources are not imported into OPA (by kube-mgmt). I can't seem to find any pattern between the resources which fail to import. So I was wondering what could be causing the issue?

[charlieegan3]

Hi @aleksandar-ruskov, my guess would be that this is related to a permissions issue. Would you be able to share the kube-mgmt logs?

[aleksandar-ruskov]

This is a part of the logs. It indeed seems like I cannot list the resources

time="2023-03-20T09:29:36Z" level=info msg="Syncing rbac.authorization.k8s.io/v1/roles." 
time="2023-03-20T09:29:36Z" level=info msg="Syncing authorization.k8s.io/v1/selfsubjectaccessreviews." 
time="2023-03-20T09:29:36Z" level=error msg="Sync for rbac.authorization.k8s.io/v1/roles failed due to OPA error. Trying again in 1s. Reason: list: roles.rbac.authorization.k8s.io is forbidden: User "system:serviceaccount:opa:default" cannot list resource "roles" in API group "rbac.authorization.k8s.io" at the cluster scope" 
time="2023-03-20T09:29:36Z" level=error msg="Sync for authorization.k8s.io/v1/selfsubjectaccessreviews failed due to OPA error. Trying again in 1s. Reason: list: selfsubjectaccessreviews.authorization.k8s.io is forbidden: User "system:serviceaccount:opa:default" cannot list resource "selfsubjectaccessreviews" in API group "authorization.k8s.io" at the cluster scope" 
time="2023-03-20T09:29:36Z" level=info msg="Syncing authorization.k8s.io/v1/localsubjectaccessreviews." 
time="2023-03-20T09:29:36Z" level=info msg="Syncing rbac.authorization.k8s.io/v1/rolebindings." 
time="2023-03-20T09:29:36Z" level=error msg="Sync for rbac.authorization.k8s.io/v1/rolebindings failed due to OPA error. Trying again in 1s. Reason: list: rolebindings.rbac.authorization.k8s.io is forbidden: User "system:serviceaccount:opa:default" cannot list resource "rolebindings" in API group "rbac.authorization.k8s.io" at the cluster scope" 

[charlieegan3]

Yeah, it looks to me like the default service account in the opa namespace doesn't have the required permissions to replicate all the resources you want.

[charlieegan3]

The k8s RBAC docs are pretty good: https://kubernetes.io/docs/reference/access-authn-authz/rbac/

You'll need to create a role for the service account with permissions to access the resources you need to replicate.