Gatekeeper Constraint not blocking the non - whitelisted image

[gviswalingam]

Hi We are using eks version 1.24 , Deployed gatekeeper via helm chart successfully into cluster.

Now added Constraint template with rego policy to allow only whitelisted images into pod as yaml file as part of helm chart. below is the code

apiVersion: templates.gatekeeper.sh/v1beta1
kind: ConstraintTemplate
metadata:
  name: k8swhitelistedimages
spec:
  crd:
    spec:
      names:
        kind: k8sWhitelistedImages
      validation:
        # Schema for the `parameters` field
        openAPIV3Schema:
          properties:
            images:
              type: array
              items: string
  targets:
    - target: admission.k8s.gatekeeper.sh
      rego: |
        package k8swhitelistedimages
        whitelisted_images = {images |
            images = input.parameters.images[_]
        }
    
        images_whitelisted(str, patterns) {
            image_matches(str, patterns[_])
        }
    
        image_matches(str, pattern) {
            contains(str, pattern)
        }
        violation[{"msg": msg}] {
          input.review.object
          image := input.review.object.spec.containers[_].image
          name := input.review.object.metadata.name
          not images_whitelisted(image, whitelisted_images)
          msg := sprintf("pod %q has invalid image %q. Please, contact your DevOps. Follow the whitelisted images %v", [name, image, whitelisted_images])
        }

After that added Constraint crd policy for whitelist images via below yaml file deployed through helm chart as well.

apiVersion: constraints.gatekeeper.sh/v1beta1
kind: k8sWhitelistedImages
metadata:
  name: k8senforcewhitelistedimages
spec:
  match:
    kinds:
      - apiGroups: [""]
        kinds: ["Pod"]
  parameters:
    images:
          - docker

Both Template and constraint were successful , verified by using the commands kubectl get constrainttemplate,constraint and also described and checked as well.

Now to test this I created a pod with image: busybox and its not whitelisted : below is the yaml file

apiVersion: v1
kind: Pod
metadata:
  name: security-context-demo
spec:
  securityContext:
    seccompProfile:
      type: RuntimeDefault
    runAsUser: 1000
    runAsGroup: 3000
    fsGroup: 2000
  volumes:
  - name: sec-ctx-vol
    emptyDir: {}
  containers:
  - name: sec-ctx-demo
    image: busybox:1.28
    command: [ "sh", "-c", "sleep 1h" ]
    volumeMounts:
    - name: sec-ctx-vol
      mountPath: /data/demo
    securityContext:
      capabilities:
        drop:
        - ALL
      allowPrivilegeEscalation: false
      runAsNonRoot: true

The pod is getting created , Actual behaviour should block this image and throw a error has invalid image Please, contact your DevOps. Follow the whitelisted image

Can someone please help me with it ?

Many thanks

[anderseknert]

The policy itself works as intended: https://play.openpolicyagent.org/p/1KGD25rK3N

Must be something missing in the configuration, or how it's set up. Not sure what that would be though.

[acpana]

hey @gviswalingam thanks for sharing your config. My TF is a little rusty and I am not gleaning the information I was looking for.

Could you please kubectl get -oyaml your gatekpeer deployments? I want to see what the flags are and their values for each gatekeeper deployment.

[gviswalingam]

Hi @acpana Please see below deployment manifests and thanks for support

apiVersion: apps/v1
kind: Deployment
metadata:
annotations:
deployment.kubernetes.io/revision: "3"
meta.helm.sh/release-name: gatekeeper-system
meta.helm.sh/release-namespace: gatekeeper-system
creationTimestamp: "2023-06-15T10:35:31Z"
generation: 3
labels:
app: gatekeeper
app.kubernetes.io/managed-by: Helm
chart: gatekeeper
control-plane: controller-manager
gatekeeper.sh/operation: webhook
gatekeeper.sh/system: "yes"
heritage: Helm
release: gatekeeper-system
name: gatekeeper-controller-manager
namespace: gatekeeper-system
resourceVersion: "4720629"
uid: 4bccfbe9-5e58-4a70-9d77-2534d320f9ab
spec:
progressDeadlineSeconds: 600
replicas: 3
revisionHistoryLimit: 10
selector:
matchLabels:
app: gatekeeper
chart: gatekeeper
control-plane: controller-manager
gatekeeper.sh/operation: webhook
gatekeeper.sh/system: "yes"
heritage: Helm
release: gatekeeper-system
strategy:
rollingUpdate:
maxSurge: 25%
maxUnavailable: 25%
type: RollingUpdate
template:
metadata:
annotations:
kubectl.kubernetes.io/restartedAt: "2023-06-15T12:35:16+01:00"
creationTimestamp: null
labels:
app: gatekeeper
chart: gatekeeper
control-plane: controller-manager
gatekeeper.sh/operation: webhook
gatekeeper.sh/system: "yes"
heritage: Helm
release: gatekeeper-system
spec:
affinity:
podAntiAffinity:
preferredDuringSchedulingIgnoredDuringExecution:
- podAffinityTerm:
labelSelector:
matchExpressions:
- key: gatekeeper.sh/operation
operator: In
values:
- webhook
topologyKey: kubernetes.io/hostname
weight: 100
automountServiceAccountToken: true
containers:
- args:
- --port=8443
- --health-addr=:9090
- --prometheus-port=8888
- --logtostderr
- --log-denies=false
- --emit-admission-events=false
- --admission-events-involved-namespace=false
- --log-level=INFO
- --exempt-namespace=gatekeeper-system
- --operation=webhook
- --enable-external-data=true
- --enable-generator-resource-expansion=false
- --log-mutations=false
- --mutation-annotations=false
- --disable-cert-rotation=false
- --max-serving-threads=-1
- --tls-min-version=1.3
- --metrics-backend=prometheus
- --operation=mutation-webhook
- --disable-opa-builtin={http.send}
command:
- /manager
env:
- name: POD_NAMESPACE
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: metadata.namespace
- name: POD_NAME
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: metadata.name
- name: NAMESPACE
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: metadata.namespace
- name: CONTAINER_NAME
value: manager
image: openpolicyagent/gatekeeper:v3.12.0
imagePullPolicy: IfNotPresent
livenessProbe:
failureThreshold: 3
httpGet:
path: /healthz
port: 9090
scheme: HTTP
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 1
name: manager
ports:
- containerPort: 8443
name: webhook-server
protocol: TCP
- containerPort: 8888
name: metrics
protocol: TCP
- containerPort: 9090
name: healthz
protocol: TCP
readinessProbe:
failureThreshold: 3
httpGet:
path: /readyz
port: 9090
scheme: HTTP
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 1
resources:
limits:
memory: 512Mi
requests:
cpu: 100m
memory: 512Mi
securityContext:
allowPrivilegeEscalation: false
capabilities:
drop:
- ALL
readOnlyRootFilesystem: true
runAsGroup: 999
runAsNonRoot: true
runAsUser: 1000
seccompProfile:
type: RuntimeDefault
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
- mountPath: /certs
name: cert
readOnly: true
dnsPolicy: ClusterFirst
nodeSelector:
kubernetes.io/os: linux
priorityClassName: system-cluster-critical
restartPolicy: Always
schedulerName: default-scheduler
securityContext:
fsGroup: 999
supplementalGroups:
- 999
serviceAccount: gatekeeper-admin
serviceAccountName: gatekeeper-admin
terminationGracePeriodSeconds: 60
volumes:
- name: cert
secret:
defaultMode: 420
secretName: gatekeeper-webhook-server-cert
status:
availableReplicas: 3
conditions:

• lastTransitionTime: "2023-06-15T10:35:34Z"
  lastUpdateTime: "2023-06-15T10:35:34Z"
  message: Deployment has minimum availability.
  reason: MinimumReplicasAvailable
  status: "True"
  type: Available
• lastTransitionTime: "2023-06-15T10:35:31Z"
  lastUpdateTime: "2023-06-15T11:35:46Z"
  message: ReplicaSet "gatekeeper-controller-manager-5f4cd96494" has successfully
  progressed.
  reason: NewReplicaSetAvailable
  status: "True"
  type: Progressing
  observedGeneration: 3
  readyReplicas: 3
  replicas: 3
  updatedReplicas: 3

gatekeeper audit deployment below

apiVersion: apps/v1
kind: Deployment
metadata:
annotations:
deployment.kubernetes.io/revision: "3"
meta.helm.sh/release-name: gatekeeper-system
meta.helm.sh/release-namespace: gatekeeper-system
creationTimestamp: "2023-06-15T10:35:31Z"
generation: 3
labels:
app: gatekeeper
app.kubernetes.io/managed-by: Helm
chart: gatekeeper
control-plane: audit-controller
gatekeeper.sh/operation: audit
gatekeeper.sh/system: "yes"
heritage: Helm
release: gatekeeper-system
name: gatekeeper-audit
namespace: gatekeeper-system
resourceVersion: "4720378"
uid: 9942432a-3e39-4017-b827-bb8c4ced28f6
spec:
progressDeadlineSeconds: 600
replicas: 1
revisionHistoryLimit: 10
selector:
matchLabels:
app: gatekeeper
chart: gatekeeper
control-plane: audit-controller
gatekeeper.sh/operation: audit
gatekeeper.sh/system: "yes"
heritage: Helm
release: gatekeeper-system
strategy:
rollingUpdate:
maxSurge: 25%
maxUnavailable: 25%
type: RollingUpdate
template:
metadata:
annotations:
kubectl.kubernetes.io/restartedAt: "2023-06-15T12:35:22+01:00"
creationTimestamp: null
labels:
app: gatekeeper
chart: gatekeeper
control-plane: audit-controller
gatekeeper.sh/operation: audit
gatekeeper.sh/system: "yes"
heritage: Helm
release: gatekeeper-system
spec:
affinity: {}
automountServiceAccountToken: true
containers:
- args:
- --audit-interval=60
- --log-level=INFO
- --constraint-violations-limit=20
- --validating-webhook-configuration-name=gatekeeper-validating-webhook-configuration
- --mutating-webhook-configuration-name=gatekeeper-mutating-webhook-configuration
- --audit-from-cache=false
- --audit-chunk-size=500
- --audit-match-kind-only=false
- --emit-audit-events=false
- --audit-events-involved-namespace=false
- --operation=audit
- --operation=status
- --operation=mutation-status
- --logtostderr
- --health-addr=:9090
- --prometheus-port=8888
- --enable-external-data=true
- --enable-generator-resource-expansion=false
- --metrics-backend=prometheus
- --disable-cert-rotation=true
command:
- /manager
env:
- name: POD_NAMESPACE
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: metadata.namespace
- name: POD_NAME
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: metadata.name
- name: NAMESPACE
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: metadata.namespace
- name: CONTAINER_NAME
value: manager
image: openpolicyagent/gatekeeper:v3.12.0
imagePullPolicy: IfNotPresent
livenessProbe:
failureThreshold: 3
httpGet:
path: /healthz
port: 9090
scheme: HTTP
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 1
name: manager
ports:
- containerPort: 8888
name: metrics
protocol: TCP
- containerPort: 9090
name: healthz
protocol: TCP
readinessProbe:
failureThreshold: 3
httpGet:
path: /readyz
port: 9090
scheme: HTTP
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 1
resources:
limits:
memory: 512Mi
requests:
cpu: 100m
memory: 512Mi
securityContext:
allowPrivilegeEscalation: false
capabilities:
drop:
- ALL
readOnlyRootFilesystem: true
runAsGroup: 999
runAsNonRoot: true
runAsUser: 1000
seccompProfile:
type: RuntimeDefault
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
- mountPath: /certs
name: cert
readOnly: true
- mountPath: /tmp/audit
name: tmp-volume
dnsPolicy: ClusterFirst
nodeSelector:
kubernetes.io/os: linux
priorityClassName: system-cluster-critical
restartPolicy: Always
schedulerName: default-scheduler
securityContext:
fsGroup: 999
supplementalGroups:
- 999
serviceAccount: gatekeeper-admin
serviceAccountName: gatekeeper-admin
terminationGracePeriodSeconds: 60
volumes:
- name: cert
secret:
defaultMode: 420
secretName: gatekeeper-webhook-server-cert
- emptyDir: {}
name: tmp-volume
status:
availableReplicas: 1
conditions:

• lastTransitionTime: "2023-06-15T10:35:36Z"
  lastUpdateTime: "2023-06-15T10:35:36Z"
  message: Deployment has minimum availability.
  reason: MinimumReplicasAvailable
  status: "True"
  type: Available
• lastTransitionTime: "2023-06-15T10:35:31Z"
  lastUpdateTime: "2023-06-15T11:35:23Z"
  message: ReplicaSet "gatekeeper-audit-5966c58bf5" has successfully progressed.
  reason: NewReplicaSetAvailable
  status: "True"
  type: Progressing
  observedGeneration: 3
  readyReplicas: 1
  replicas: 1
  updatedReplicas: 1

[gviswalingam]

@anderseknert @acpana . The same policy blocks images/repos in other namespaces apart from default and gate-keeper-system.
Any idea what change required to fix this guys ?

Thanks

[gviswalingam]

Removing ignore label from default namespace resolves the issue and namespace where admission controller is deployed, policy wont work/applied in that namespace.