Process PDB policies

[xzizka]

Hello,

I deployed Trivy Operator 0.20.6 with Trivy 0.18.5 on K8S 1.29. So far, so good.
Now, I want to implement my own policy. The policy looks like this:

  policy.pdbtest.kinds: '*'
  policy.pdbtest.rego: |
    package trivyoperator.policy.k8s.pdbtest

    import data.lib.result
    import future.keywords.in

    __rego_metadata__ := {
        "id": "pdb_detection",
        "title": "PDB detection labels",
        "severity": "CRITICAL",
        "type": "Kubernetes Configuration Check",
        "description": "Pod Disruption Budgets may prevent the node drain if minAvailable value is equal to number of running pods.",
        "recommended_actions": "Number of running pods must be bigger than the minAvailable value of Pod Disruption Budgets.",
        "url": "https://kubernetes.io/docs/tasks/run-application/configure-pdb/#think-about-how-your-application-reacts-to-disruptions",
    }

    deny[res] {
        input.kind == "PodDisruptionBudget"
        min_available := input.spec.minAvailable
        currently_healthy := input.status.currentHealthy
        gap := currently_healthy - min_available
        gap <= 0
        msg := sprintf("PDB value of minAvailable is equal of number of running pods: %v", [currently_healthy])
        res := {"msg": msg}
    }

It is located in configmap trivy-operator-policies-config. Unfortunatelly, this policy hasn't beeing processed. I have one PDB that should be detected by this policy:

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  annotations:
    operator-sdk/primary-resource: kube-system/app-monitoring
    operator-sdk/primary-resource-type: K8SComponentsInstaller.apps.k8s
  creationTimestamp: "2024-03-28T18:19:02Z"
  generation: 1
  labels:
    app.kubernetes.io/instance: monitoring
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: telegraf-prometheus-adapter
    helm.sh/chart: telegraf-prometheus-adapter-1.1.2
  name: monitoring-telegraf-prometheus-adapter
  namespace: monitoring-system
  resourceVersion: "17498"
  uid: 5bb4db09-7472-4fad-8bad-885f0cf6d38d
spec:
  minAvailable: 1
  selector:
    matchLabels:
      app.kubernetes.io/instance: monitoring
      app.kubernetes.io/name: telegraf-prometheus-adapter
status:
  conditions:
  - lastTransitionTime: "2024-03-28T18:19:11Z"
    message: ""
    observedGeneration: 1
    reason: InsufficientPods
    status: "False"
    type: DisruptionAllowed
  currentHealthy: 1
  desiredHealthy: 1
  disruptionsAllowed: 0
  expectedPods: 1
  observedGeneration: 1

If I run the command manually, it is able to detect the PDB and process the policy:

$ trivy k8s --trace --db-repository <dockerhub>/iao-sq013-pub/aquasecurity/trivy-db:2 --skip-policy-update --debug --report all --config-policy policies --policy-namespaces trivyoperator --cache-dir /tmp --all-namespaces PodDisruptionBudget/
2024-04-02T14:37:07.989+0200    DEBUG   Severities: ["UNKNOWN" "LOW" "MEDIUM" "HIGH" "CRITICAL"]
2024-04-02T14:37:07.991+0200    DEBUG   Ignore statuses {"statuses": null}
2024-04-02T14:37:08.243+0200    DEBUG   cache dir:  /tmp
2024-04-02T14:37:08.243+0200    DEBUG   DB update was skipped because the local DB was downloaded during the last hour
2024-04-02T14:37:08.243+0200    DEBUG   DB Schema: 2, UpdatedAt: 2024-01-08 12:12:18.932722798 +0000 UTC, NextUpdate: 2024-01-08 18:12:18.932722417 +0000 UTC, DownloadedAt: 2024-04-02 11:52:40.00750863 +0000 UTC
6 / 6 [----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------] 100.00% 0 p/s

namespace: monitoring-system, poddisruptionbudget: monitoring-telegraf-prometheus-adapter (kubernetes)

Tests: 106 (SUCCESSES: 105, FAILURES: 1, EXCEPTIONS: 0)
Failures: 1 (UNKNOWN: 0, LOW: 0, MEDIUM: 0, HIGH: 0, CRITICAL: 1)

CRITICAL: PDB value of minAvailable is equal of number of running pods: 1
════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════
Pod Disruption Budgets may prevent the node drain if minAvailable value is equal to number of running pods.

See https://kubernetes.io/docs/tasks/run-application/configure-pdb/#think-about-how-your-application-reacts-to-disruptions
────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────



namespace: kube-system, poddisruptionbudget: coredns (kubernetes)

Tests: 106 (SUCCESSES: 105, FAILURES: 1, EXCEPTIONS: 0)
Failures: 1 (UNKNOWN: 0, LOW: 0, MEDIUM: 1, HIGH: 0, CRITICAL: 0)

MEDIUM: PodDisruptionBudget 'coredns' should not be set with 'kube-system' namespace
════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════
ensure that User pods are not placed in kube-system namespace

See https://avd.aquasec.com/misconfig/ksv037
────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
 namespace: kube-system, poddisruptionbudget: coredns:8-11
────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
   8 ┌     maxUnavailable: 30%
   9 │     selector:
  10 │         matchLabels:
  11 └             k8s-app: kube-dns
────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────



namespace: kube-system, poddisruptionbudget: cinder-csi-pdb (kubernetes)

Tests: 106 (SUCCESSES: 105, FAILURES: 1, EXCEPTIONS: 0)
Failures: 1 (UNKNOWN: 0, LOW: 0, MEDIUM: 1, HIGH: 0, CRITICAL: 0)

MEDIUM: PodDisruptionBudget 'cinder-csi-pdb' should not be set with 'kube-system' namespace
════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════
ensure that User pods are not placed in kube-system namespace

See https://avd.aquasec.com/misconfig/ksv037
────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
 namespace: kube-system, poddisruptionbudget: cinder-csi-pdb:8-11
────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
   8 ┌     minAvailable: 0
   9 │     selector:
  10 │         matchLabels:
  11 └             app: csi-cinder-controllerplugin
────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────

But this policy does not work when is part of the Trivy Operator configMap trivy-operator-policies-config.
Here is ConfigMap trivy-operator-trivy-config:

  trivy.additionalVulnerabilityReportFields: ""
  trivy.command: rootfs
  trivy.dbRepository: <dockerhub>/ghcr/aquasecurity/trivy-db
  trivy.dbRepositoryInsecure: "true"
  trivy.filesystemScanCacheDir: /var/trivyoperator/trivy-db
  trivy.imagePullPolicy: IfNotPresent
  trivy.imagePullSecret: dockerhub
  trivy.imageScanCacheDir: /tmp/trivy/.cache
  trivy.includeDevDeps: "false"
  trivy.javaDbRepository: <dockerhub>/ghcr/aquasecurity/trivy-java-db
  trivy.mode: Standalone
  trivy.offlineScan: "true"
  trivy.repository: <dockerhub>/iao-sq013-pub/aquasecurity/trivy
  trivy.resources.limits.cpu: 500m
  trivy.resources.limits.memory: 1G
  trivy.resources.requests.cpu: 100m
  trivy.resources.requests.memory: 512M
  trivy.sbomSources: ""
  trivy.severity: UNKNOWN,LOW,MEDIUM,HIGH,CRITICAL
  trivy.skipJavaDBUpdate: "false"
  trivy.slow: "true"
  trivy.supportedConfigAuditKinds: Workload,Service,Role,ClusterRole,NetworkPolicy,Ingress,LimitRange,ResourceQuota,PodDisruptionBudget
  trivy.tag: 0.49.1
  trivy.timeout: 5m0s
  trivy.useBuiltinRegoPolicies: "true"

When I check the trivy-operator pod logs I see there controllers for e.g. ingress or resourcequota, but not for pdb:

$ kubectl logs trivy-operator-6f4fb8466f-kk2c2 -n trivy-system | grep -i ingress
{"level":"info","ts":"2024-04-02T14:01:35Z","msg":"Starting EventSource","controller":"ingress","controllerGroup":"networking.k8s.io","controllerKind":"Ingress","source":"kind source: *v1.Ingress"}
{"level":"info","ts":"2024-04-02T14:01:35Z","msg":"Starting EventSource","controller":"ingress","controllerGroup":"networking.k8s.io","controllerKind":"Ingress","source":"kind source: *v1alpha1.ConfigAuditReport"}
{"level":"info","ts":"2024-04-02T14:01:35Z","msg":"Starting EventSource","controller":"ingress","controllerGroup":"networking.k8s.io","controllerKind":"Ingress","source":"kind source: *v1alpha1.InfraAssessmentReport"}
{"level":"info","ts":"2024-04-02T14:01:35Z","msg":"Starting Controller","controller":"ingress","controllerGroup":"networking.k8s.io","controllerKind":"Ingress"}
{"level":"info","ts":"2024-04-02T14:01:38Z","msg":"Starting workers","controller":"ingress","controllerGroup":"networking.k8s.io","controllerKind":"Ingress","worker count":1}
e57toc@vncub17999:~ $ kubectl logs trivy-operator-6f4fb8466f-kk2c2 -n trivy-system | grep -i quota
2024/04/02 14:01:34 maxprocs: Leaving GOMAXPROCS=4: CPU quota undefined
{"level":"info","ts":"2024-04-02T14:01:35Z","msg":"Starting EventSource","controller":"resourcequota","controllerGroup":"","controllerKind":"ResourceQuota","source":"kind source: *v1.ResourceQuota"}
{"level":"info","ts":"2024-04-02T14:01:35Z","msg":"Starting EventSource","controller":"resourcequota","controllerGroup":"","controllerKind":"ResourceQuota","source":"kind source: *v1alpha1.ConfigAuditReport"}
{"level":"info","ts":"2024-04-02T14:01:35Z","msg":"Starting Controller","controller":"resourcequota","controllerGroup":"","controllerKind":"ResourceQuota"}
{"level":"info","ts":"2024-04-02T14:01:38Z","msg":"Starting workers","controller":"resourcequota","controllerGroup":"","controllerKind":"ResourceQuota","worker count":1}
$ kubectl logs trivy-operator-6f4fb8466f-kk2c2 -n trivy-system | grep -i budget
$

Am I missing any configuration? How to set the trivy operator to also process policies for PDBs?
Thank you

[xzizka]

One more point.... In inserted into the trivy-operator-policies-config configmap BAR and FOO policies.

apiVersion: v1
data:
  policy.bar.kinds: '*'
  policy.bar.rego: |
    package trivyoperator.policy.k8s.bar

    import future.keywords
    #import data.lib.result

    __rego_metadata__ := {
        "id": "BAR memory_limits",
        "title": "BAR Memory limits required",
        "severity": "CRITICAL",
        "description": "BAR The usage of memory limits is required.",
    }

    __rego_input__ := {
        "selector": [
            {"type": "kubernetes"},
        ],
    }

    deny[res] {
        input.kind == "ReplicaSet"
        some container in input.spec.template.spec.containers
        not container.resources.limits.memory
        msg := sprintf("Container '%v/%v' does not have memory limits", [input.metadata.name, container.name])
        res := {"msg": msg}
    }
  policy.foo.kinds: '*'
  policy.foo.rego: |
    package trivyoperator.policy.k8s.foo

    __rego_metadata__ := {
        "id": "FOO recommended_labels",
        "title": "FOO Recommended labels",
        "severity": "LOW",
        "type": "Kubernetes Security Check",
        "description": "A common set of labels allows tools to work interoperably, describing objects in a common manner that all tools can understand.",
        "recommended_actions": "Take full advantage of using recommended labels and apply them on every resource object.",
        "url": "https://kubernetes.io/docs/concepts/overview/working-with-objects/common-labels/",
    }

    recommended_labels := [
        "app.kubernetes.io/name",
        "app.kubernetes.io/version",
    ]

    deny[res] {
        provided := {label | input.metadata.labels[label]}
        required := {label | label := recommended_labels[_]}
        missing := required - provided
        count(missing) > 0
        msg := sprintf("You must provide labels: %v", [missing])
        res := {"msg": msg}
    }
  policy.pdbtest.kinds: '*'
  policy.pdbtest.rego: |
    package trivyoperator.policy.k8s.pdbtest

    import data.lib.result
    import future.keywords.in

    __rego_metadata__ := {
        "id": "pdb_detection",
        "title": "PDB detection labels",
        "severity": "CRITICAL",
        "type": "Kubernetes Configuration Check",
        "description": "Pod Disruption Budgets may prevent the node drain if minAvailable value is equal to number of running pods.",
        "recommended_actions": "Number of running pods must be bigger than the minAvailable value of Pod Disruption Budgets.",
        "url": "https://kubernetes.io/docs/tasks/run-application/configure-pdb/#think-about-how-your-application-reacts-to-disruptions",
    }

    deny[res] {
        input.kind == "PodDisruptionBudget"
        min_available := input.spec.minAvailable
        currently_healthy := input.status.currentHealthy
        gap := currently_healthy - min_available
        gap <= 0
        msg := sprintf("PDB value of minAvailable is equal of number of running pods: %v", [currently_healthy])
        res := {"msg": msg}
    }
kind: ConfigMap
metadata:
  annotations:
    operator-sdk/primary-resource: kube-system/app-trivy
    operator-sdk/primary-resource-type: K8SComponentsInstaller.apps.k8s.kb.cz
  creationTimestamp: "2024-03-28T18:17:32Z"
  labels:
    app.kubernetes.io/instance: trivy-operator
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: trivy-operator
    app.kubernetes.io/version: 0.18.5
    helm.sh/chart: trivy-operator-0.20.6
  name: trivy-operator-policies-config
  namespace: trivy-system
  resourceVersion: "2307846"
  uid: b1520f7d-cf89-477a-b50c-b32ee0a634b3

I see the FOO and BAR results:

$ kg configauditreports.aquasecurity.github.io -A -o yaml | grep -e BAR -e FOO
      checkID: FOO recommended_labels
      title: FOO Recommended labels
      checkID: FOO recommended_labels
      title: FOO Recommended labels
      checkID: FOO recommended_labels
      title: FOO Recommended labels
      checkID: FOO recommended_labels
      title: FOO Recommended labels
      checkID: FOO recommended_labels
      title: FOO Recommended labels
      checkID: FOO recommended_labels
      title: FOO Recommended labels
      checkID: FOO recommended_labels
      title: FOO Recommended labels
      checkID: BAR memory_limits
      description: BAR The usage of memory limits is required.
      title: BAR Memory limits required
...

But I don't see the PDBTEST result.

$ kg configauditreports.aquasecurity.github.io -A -o yaml | grep -i PDB
$

It looks like Trivy does not read the 'PodDisruptionBudget' even if it is in the configured Kinds:

trivy.supportedConfigAuditKinds: Workload,Service,Role,ClusterRole,NetworkPolicy,Ingress,LimitRange,ResourceQuota,PodDisruptionBudget

Is there a way how to configure trivy-operator to read PodDisruptionBudget kind?

[xzizka]

Regarding the discussion with @chen-keinan only the following kinds are supported:

Workload,Service,Role,ClusterRole,NetworkPolicy,Ingress,LimitRange,ResourceQuota

Currently, there are no plans to add any other kinds to this list.

Thank you @chen-keinan for clarification.

[lsoica]

So If I want to add a policy to validate that each statefulset is targeted by a PDB, there is no way to do it with the operator ?

[xzizka]

Currently, there is no way.

[chen-keinan]

@xzizka @lsoica we do not object the idea, the challenge is to support it and still follow the operator pattern, resource reconciliation , any ideas are welcome

[lsoica]

Understood. I can't speak for other use cases, but in my situation, the operator pattern doesn't provide any intrinsic value. My main goal is to perform automated continuous/periodic scans at runtime against a set of predefined best practices, and then make the results available for visualization and alerting through a system like Grafana. Whether or not an operator is used doesn't really matter to me.

What would be the shortcomings that prevent the use of the operator pattern in this context?

[chen-keinan]

the operator need to statically know in advance which resources to reconcile, making it dynamically is the issue