Gatekeeper logs not showing violations

[gviswalingam]

Hi

I have deployed gatekeeper v3.12.0 via helm chart and constraint template and crd to allow only the whitelisted repos. the policy works fine and blocks the pod from getting created, if I use non-whitelisted image/repo and we get the error message .I also see the violations for the deployments of existing images/repos prior to installing gatekeeper in audit logs, which is not whitelisted in policy. Once we add them in policy those violation disappear in audit logs. But when I try to deploy a pod with non-whitelisted image/repo, gatekeeper blocks it but I don't see the details of violation in logs , why is that ?
Below is some of the logs
│ gatekeeper-audit-679cf8789c-8jbs2 {"level":"info","ts":1687520658.3459783,"logger":"controller","msg":"auditing constraints and violations","process":"audit","audit_id":"2023-06-23T11:44:18Z","event_type":"audit_started"} ││ gatekeeper-audit-679cf8789c-8jbs2 {"level":"info","ts":1687520659.396397,"msg":"Waited for 1.040206619s due to client-side throttling, not priority and fairness, request: GET:https://10.100.0.1:443/apis/coordination.k8s.io/v1?timeout=32s\n"} ││ gatekeeper-audit-679cf8789c-8jbs2 {"level":"info","ts":1687520659.4044647,"logger":"controller","msg":"Auditing via discovery client","process":"audit","audit_id":"2023-06-23T11:44:18Z"} ││ gatekeeper-audit-679cf8789c-8jbs2 {"level":"info","ts":1687520660.606831,"logger":"controller","msg":"closing the previous audit reporting thread","process":"audit","audit_id":"2023-06-23T11:44:18Z"} ││ gatekeeper-audit-679cf8789c-8jbs2 {"level":"info","ts":1687520660.6068585,"logger":"controller","msg":"auditing is complete","process":"audit","audit_id":"2023-06-23T11:44:18Z","event_type":"audit_finished"} ││ gatekeeper-audit-679cf8789c-8jbs2 {"level":"info","ts":1687520660.6068735,"logger":"controller","msg":"constraint","process":"audit","audit_id":"2023-06-23T11:44:18Z","resource kind":"K8sAllowedRepos"} ││ gatekeeper-audit-679cf8789c-8jbs2 {"level":"info","ts":1687520660.610798,"logger":"controller","msg":"constraint","process":"audit","audit_id":"2023-06-23T11:44:18Z","count of constraints":1} ││ gatekeeper-audit-679cf8789c-8jbs2 {"level":"info","ts":1687520660.610829,"logger":"controller","msg":"starting update constraints loop","process":"audit","audit_id":"2023-06-23T11:44:18Z","constraints to update":"map[{constraints.gatekeeper.sh K8sAllowedRepos v1beta1 allow-onl ││ gatekeeper-audit-679cf8789c-8jbs2 {"level":"info","ts":1687520660.615326,"logger":"controller","msg":"updating constraint status","process":"audit","audit_id":"2023-06-23T11:44:18Z","constraintName":"allow-only-private-registry"} ││ gatekeeper-audit-679cf8789c-8jbs2 {"level":"info","ts":1687520660.6249003,"logger":"controller","msg":"handling constraint update","process":"constraint_controller","instance":{"apiVersion":"constraints.gatekeeper.sh/v1beta1","kind":"K8sAllowedRepos","name":"allow-only-private- │

Can someone help please ?

Thanks

[maxsmythe]

Audit only scans/logs violations for existing resources on the cluster. Something that gets rejected by the webhook wouldn't be persisted in the cluster and therefore wouldn't show up in audit logs.

If you want to see webhook rejections in the webhook pod logs (not the audit pod), setting the logDenies Helm chart parameter would do that. Do note that this may significantly increase the volume of logs written by the webhook pod.

[gviswalingam]

Thanks, It make sense .

Just for clarifications , when you say webhook pod, you mean the gatekeeper control manager pod right?
Also If I modify logDenies as true in value.yaml that should work right ?

Thanks again

[gviswalingam]

Hi @maxsmythe I tried enabling logdenies and tested if i can see logs in control manager pods . I only get the below logs

{"level":"info","ts":1687763531.108483,"logger":"controller","msg":"loading code into OPA","kind":"ConstraintTemplate","process":"constraint_template_controller","name":"k8sallowedrepos","crdName":"k8sallowedrepos.constraints.gatekeeper.sh"} ││ {"level":"info","ts":1687763531.1085532,"logger":"controller","msg":"[readiness] observed ConstraintTemplate","kind":"ConstraintTemplate","process":"constraint_template_controller","name":"k8sallowedrepos","crdName":"k8sallowedrepos.constraints.gatekeeper.sh","name":"k8sallowedrepos"} ││ {"level":"info","ts":1687763531.1088097,"logger":"controller","msg":"making sure constraint is in watcher registry","kind":"ConstraintTemplate","process":"constraint_template_controller","name":"k8sallowedrepos","crdName":"k8sallowedrepos.constraints.gatekeeper.sh"} ││ {"level":"info","ts":1687763531.1193373,"logger":"controller","msg":"template was updated","kind":"ConstraintTemplate","process":"constraint_template_controller","event_type":"template_updated","template_name":"k8sallowedrepos"} ││ {"level":"info","ts":1687763531.12511,"logger":"controller","msg":"handling constraint update","process":"constraint_controller","instance":{"apiVersion":"constraints.gatekeeper.sh/v1beta1","kind":"K8sAllowedRepos","name":"allow-only-private-registry"}} ││ 2023/06/26 07:12:15 http: TLS handshake error from 172.20.192.59:51968: EOF ││ 2023/06/26 07:12:20 http: TLS handshake error from 172.20.192.59:51974: EOF ││ {"level":"info","ts":1687763540.647167,"logger":"controller","msg":"handling constraint update","process":"constraint_controller","instance":{"apiVersion":"constraints.gatekeeper.sh/v1beta1","kind":"K8sAllowedRepos","name":"allow-only-private-registry"}} ││ 2023/06/26 07:12:21 http: TLS handshake error from 172.20.242.242:45320: EOF ││ {"level":"info","ts":1687763542.2776103,"logger":"controller","msg":"handling constraint update","process":"constraint_controller","instance":{"apiVersion":"constraints.gatekeeper.sh/v1beta1","kind":"K8sAllowedRepos","name":"allow-only-private-registry"}} ││ {"level":"info","ts":1687763542.2780592,"logger":"controller","msg":"[readiness] observed Constraint","process":"constraint_controller","name":"allow-only-private-registry"} ││ {"level":"info","ts":1687763542.278077,"logger":"controller","msg":"constraint added to OPA","process":"constraint_controller","event_type":"constraint_added","constraint_group":"constraints.gatekeeper.sh","constraint_api_version":"v1beta1","constraint_kind":"K8sAllowedRepos","constraint_name":"allow-only-private-registry","constraint_action":"deny","constraint_status":"enforced"} ││ {"level":"info","ts":1687763542.2897377,"logger":"controller","msg":"handling constraint update","process":"constraint_controller","instance":{"apiVersion":"constraints.gatekeeper.sh/v1beta1","kind":"K8sAllowedRepos","name":"allow-only-private-registry"}} ││ {"level":"info","ts":1687763542.3080206,"logger":"controller","msg":"handling constraint update","process":"constraint_controller","instance":{"apiVersion":"constraints.gatekeeper.sh/v1beta1","kind":"K8sAllowedRepos","name":"allow-only-private-registry"}} ││ {"level":"info","ts":1687763542.3200624,"logger":"controller","msg":"handling constraint update","process":"constraint_controller","instance":{"apiVersion":"constraints.gatekeeper.sh/v1beta1","kind":"K8sAllowedRepos","name":"allow-only-private-registry"}} ││ 2023/06/26 07:12:30 http: TLS handshake error from 172.20.192.59:55186: EOF

[maxsmythe]

Correct, the logs should be in the controller manager pod (there are 3 of them by default, so any given pod may not have the log line you're looking for).

WRT not being able to see the logs:

1. Have there been any requests to the cluster that have been denied since the pods were restarted with the new arguments?

2. can you show what the arguments to the controller-manager pod is? (kubectl get pod -o yaml for the controller-manager pods)

[gviswalingam]

apiVersion: v1
items:

• apiVersion: v1
  kind: Pod
  metadata:
  annotations:
  container.seccomp.security.alpha.kubernetes.io/app: runtime/default
  kubectl.kubernetes.io/last-applied-configuration: |
  {"apiVersion":"v1","kind":"Pod","metadata":{"annotations":{},"name":"app","namespace":"default"},"spec":{"containers":[{"args":["-c","while true; do echo $(date -u) \u003e\u003e /data/out.txt; sleep 5; done"],"command":["/bin/sh"],"image":"reireias/non-root-user-ubuntu","name":"app","securityContext":{"allowPrivilegeEscalation":false,"capabilities":{"drop":["ALL"]},"runAsNonRoot":true,"runAsUser":1000,"seccompProfile":{"type":"RuntimeDefault"}},"volumeMounts":[{"mountPath":"/data","name":"persistent-storage"}]}],"volumes":[{"name":"persistent-storage","persistentVolumeClaim":{"claimName":"ebs-claim"}}]}}
  kubernetes.io/psp: aws-node-termination-handler
  creationTimestamp: "2023-06-22T07:24:19Z"
  name: app
  namespace: default
  resourceVersion: xxxxxxx
  uid: xxxxx
  spec:
  containers:
  • args:
    • -c
    • while true; do echo $(date -u) >> /data/out.txt; sleep 5; done
      command:
    • /bin/sh
      image: reireias/non-root-user-ubuntu
      imagePullPolicy: Always
      name: app
      resources: {}
      securityContext:
      allowPrivilegeEscalation: false
      capabilities:
      drop:
      • ALL
        runAsNonRoot: true
        runAsUser: 1000
        seccompProfile:
        type: RuntimeDefault
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
    • mountPath: /data
      name: persistent-storage
    • mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: kube-api-access-7vskd
      readOnly: true
      dnsPolicy: ClusterFirst
      enableServiceLinks: true
      nodeName: ip-172-20-215-93.eu-west-1.compute.internal
      preemptionPolicy: PreemptLowerPriority
      priority: 100000
      priorityClassName: wcc-global-default-priorityclass
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext: {}
      serviceAccount: default
      serviceAccountName: default
      terminationGracePeriodSeconds: 30
      tolerations:
  • effect: NoExecute
    key: node.kubernetes.io/not-ready
    operator: Exists
    tolerationSeconds: 300
  • effect: NoExecute
    key: node.kubernetes.io/unreachable
    operator: Exists
    tolerationSeconds: 300
    volumes:
  • name: persistent-storage
    persistentVolumeClaim:
    claimName: ebs-claim
  • name: kube-api-access-xxxx
    projected:
    defaultMode: 420
    sources:
    • serviceAccountToken:
      expirationSeconds: 3607
      path: token
    • configMap:
      items:
      • key: ca.crt
        path: ca.crt
        name: kube-root-ca.crt
    • downwardAPI:
      items:
      • fieldRef:
        apiVersion: v1
        fieldPath: metadata.namespace
        path: namespace
        status:
        conditions:
  • lastProbeTime: null
    lastTransitionTime: "2023-06-22T07:24:25Z"
    status: "True"
    type: Initialized
  • lastProbeTime: null
    lastTransitionTime: "2023-06-22T07:24:30Z"
    status: "True"
    type: Ready
  • lastProbeTime: null
    lastTransitionTime: "2023-06-22T07:24:30Z"
    status: "True"
    type: ContainersReady
  • lastProbeTime: null
    lastTransitionTime: "2023-06-22T07:24:25Z"
    status: "True"
    type: PodScheduled
    containerStatuses:
  • containerID: containerd://xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
    image: docker.io/reireias/non-root-user-ubuntu:latest
    imageID: docker.io/reireias/non-root-user-ubuntu@sha256:xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
    lastState: {}
    name: app
    ready: true
    restartCount: 0
    started: true
    state:
    running:
    startedAt: "2023-06-22T07:24:29Z"
    hostIP: 172..xx.xx.xxx
    phase: Running
    podIP: 172.xx.xxx.xxx
    podIPs:
  • ip: 172.xx.xxx.xxx
    qosClass: BestEffort
    startTime: "2023-06-22T07:24:25Z"
    kind: List
    metadata:
    resourceVersion: ""

[maxsmythe]

I'm not seeing the Gatekeeper docker image. Also not seeing the args provided to the Gatekeeper command (which is the main thing I wanted to verify).