OPA Gatekeeper dashboard

[xamroc]

I’m reading a lot of information out there on how to display OPA Gatekeeper violations on a dashboard. Some are using prometheus to scrape from audit, controller-manager or writing their own exporters. How is everyone here implementing OPA Gatekeeper dashboards? Any recommendations?

[srenatus]

Gatekeeper Policy Manager looks relevant here, have you seen it before? I've never really dug into using it, though, but I'm sure others have and could share their experiences here.

[xamroc]

Very cool! Will definitely have a look. However, I'd prefer to use one common tool for monitoring instead of introducing another one. One example is that almost everyone has a Prometheus/Grafana monitoring their clusters. Based on everyone's experience, are there pitfalls in this approach over adopting a specialised tool?

[srenatus]

Sorry I had misunderstood the intent of your question a bit. 😅 I'm afraid I don't have enough experience operating GK to give a more useful answer here.

[maxsmythe]

how to display OPA Gatekeeper violations on a dashboard

One important thing to keep in mind is how thorough you want the reporting to be.

• Prometheus will give aggregated information, like # of violations, but because Prometheus metrics are, by default, insecure, and because they have limits in terms of how high-cardinality they can be, there is no fine-grained detail
• The status field of a constraint will report fine-grained violations, but it has limits:
  • It only reports violations for the most recent audit run
  • It will report up to X violations (where X is configurable via flag), after which violations are no longer reported on the status field of a constraint. This is because K8s resources have an upper-limit as to how large they can be.
• Pod logs have the most detailed information -- all violations should be reported in the pod logs, and because pod logs can be persisted, it's possible to get a view of historical violations. The downside is that scraping pod logs is more involved.

[jalseth]

In a previous role, we went the route of parsing audit logs for this reason, which turned out to be critical in our case since some policies had many violations and we wouldn't have been able to get a complete picture using just the status of the Constraints due to the limit of the violations. It's also easier since our logging platform has built in dashboarding capabilities, which you don't get out of the box by querying the status of the resources in K8s.

[MarkVLK]

@maxsmythe noob question, but related to your point of pod logs having the most detailed information (including OPA Gatekeeper violations), do you have an example of where/how one can retrieve these pod logs or if they're emitted by default?

I'm trying to find constraint status, including current active violations, but have been unable to find them in logs thus far and not sure if I'm looking in the right place or perhaps don't have the proper flags set, or if there's a better way to identify active and historical constraint violations.

[maxsmythe]

Sorry, this got buried in my stack.

I don't think constraint status currently shows up in logs, but audit results do.

This file has good keywords to search for in the logs:

https://github.com/open-policy-agent/gatekeeper/blob/92841ffea3b170e19c1ac24232d9e424d688b141/pkg/logging/logging.go#L3-L28

[maxsmythe]

The audit pod would be where to look for audit logs.