Moving to "admissionregistration.k8s.io/v1" causes OPA to not work

[kylegoch]

We are prepping to move from Kubernetes 1.21 to 1.22 and need to move our OPA Mutating Webhook over to admissionregistration.k8s.io/v1. However when we make the update, OPA stops working.

We use OPA Mutating Webhook to add an annotation to new namespaces. With admissionregistration.k8s.io/v1beta1 this works fine. But updating to admissionregistration.k8s.io/v1 the namespace does not get annotated. However, I can see a patch issued that contains the annotation in the OPA logs. But the annotation never shows up.

Working Webhook Configuration:

kind: MutatingWebhookConfiguration
apiVersion: admissionregistration.k8s.io/v1beta1
metadata:
  name: aaa-opa-mutating-webhook
  annotations:
    cert-manager.io/inject-ca-from: opa/opa-server
webhooks:
  - name: mutating-webhook.openpolicyagent.org
    namespaceSelector:
      matchExpressions:
      - key: openpolicyagent.org/webhook
        operator: NotIn
        values:
        - ignore
    rules:
      - operations: ["*"]
        apiGroups: ["*"]
        apiVersions: ["*"]
        resources: ["*"]
    clientConfig:
      service:
        namespace: opa
        name: opa

Not Working Webhook Configuration:

kind: MutatingWebhookConfiguration
apiVersion: admissionregistration.k8s.io/v1
metadata:
  name: aaa-opa-mutating-webhook
  annotations:
    cert-manager.io/inject-ca-from: opa/opa-server
webhooks:
  - name: mutating-webhook.openpolicyagent.org
    admissionReviewVersions: ["v1"]
    sideEffects: None
    namespaceSelector:
      matchExpressions:
      - key: openpolicyagent.org/webhook
        operator: NotIn
        values:
        - ignore
    rules:
      - operations: ["*"]
        apiGroups: ["*"]
        apiVersions: ["*"]
        resources: ["*"]
    clientConfig:
      service:
        namespace: opa
        name: opa

Here is the OPA deployment YAML

# Creates the service account for OPA
apiVersion: v1
kind: ServiceAccount
metadata:
  namespace: opa
  name: opa
---
# Grant OPA/kube-mgmt read-only access to resources. This lets kube-mgmt
# replicate resources into OPA so they can be used in policies.
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: opa-viewer
roleRef:
  kind: ClusterRole
  name: view
  apiGroup: rbac.authorization.k8s.io
subjects:
- kind: ServiceAccount
  name: opa
  namespace: opa
---
# Define role for OPA/kube-mgmt to update configmaps with policy status.
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  namespace: opa
  name: configmap-modifier
rules:
- apiGroups: [""]
  resources: ["configmaps"]
  verbs: ["update", "patch"]
---
# Grant OPA/kube-mgmt role defined above.
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  namespace: opa
  name: opa-configmap-modifier
roleRef:
  kind: Role
  name: configmap-modifier
  apiGroup: rbac.authorization.k8s.io
subjects:
- kind: ServiceAccount
  name: opa
  namespace: opa
---
kind: Service
apiVersion: v1
metadata:
  name: opa
  namespace: opa
spec:
  selector:
    app: opa
  ports:
  - name: https
    protocol: TCP
    port: 443
    targetPort: 443
---
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: opa
  namespace: opa
  name: opa
spec:
  replicas: 3
  selector:
    matchLabels:
      app: opa
  template:
    metadata:
      labels:
        app: opa
      name: opa
    spec:
      containers:
        # WARNING: OPA is NOT running with an authorization policy configured. This
        # means that clients can read and write policies in OPA. If you are
        # deploying OPA in an insecure environment, be sure to configure
        # authentication and authorization on the daemon. See the Security page for
        # details: https://www.openpolicyagent.org/docs/security.html.
        - name: opa
          image: openpolicyagent/opa:latest
          args:
            - "run"
            - "--server"
            - "--tls-cert-file=/certs/tls.crt"
            - "--tls-private-key-file=/certs/tls.key"
            - "--addr=0.0.0.0:443"
            - "--addr=http://127.0.0.1:8181"
            - "--log-format=json"
            - "--log-level=error"
            - "--set=decision_logs.console=true"
            - "--set=default_decision=/kube/mutation/main"
          volumeMounts:
            - readOnly: true
              mountPath: /certs
              name: opa-server
          readinessProbe:
            httpGet:
              path: /health?plugins&bundle
              scheme: HTTPS
              port: 443
            initialDelaySeconds: 3
            periodSeconds: 5
          livenessProbe:
            httpGet:
              path: /health
              scheme: HTTPS
              port: 443
            initialDelaySeconds: 3
            periodSeconds: 5
        - name: kube-mgmt
          image: openpolicyagent/kube-mgmt:2.0.1
          args:
            - "--replicate-cluster=v1/namespaces"
            - "--replicate=networking.k8s.io/v1/ingresses"
      volumes:
        - name: opa-server
          secret:
            secretName: opa-server
      serviceAccountName: opa
      affinity:
        podAntiAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
            - weight: 100
              podAffinityTerm:
                labelSelector:
                  matchExpressions:
                    - key: app
                      operator: In
                      values:
                        - opa
                topologyKey: "kubernetes.io/hostname"
      tolerations:
        - effect: NoExecute
          operator: Exists
        - effect: NoSchedule
          operator: Exists
        - effect: PreferNoSchedule
          operator: Exists
      nodeSelector:
        kubernetes.io/role: master

Let me know what else may be helpful. Currently we have rolled back to v1beta1 for the Webhook so that it will continue to work, but I'm scratching my head here.

[anderseknert]

What version of the admission review object does your policy respond with? I'm going to guess that this setting:

admissionReviewVersions: ["v1"]

Is going to have the Kubernetes API both send and expect v1 versioned admission reviews only, so unless that's what you're sending back they'll likely be rejected.

[kylegoch]

This was it, thanks! I had not updated my admissionReview apiVersion 🤦 . Brought that up to v1 and that got everything working again.

[anderseknert]

Perfect! :)