config: enable cilium service monitors by default

[Mlundm]

Warning

This is a public repository, ensure not to disclose:

• personal data beyond what is necessary for interacting with this pull request, nor
• business confidential information, such as customer names.

What kind of PR is this?

This enables the service monitors for Cilium by default.

These are needed for the Cilium grafana dashboards to work and so I thought it would be nice and make sense to have these enabled by default.

CAPI PR: https://github.com/elastisys/ck8s-cluster-api/pull/436

Required: Mark one of the following that is applicable:

• kind/feature
• kind/improvement
• kind/deprecation
• kind/documentation
• kind/clean-up
• kind/bug
• kind/other

Optional: Mark one or more of the following that are applicable:

Important

Breaking changes should be marked kind/admin-change or kind/dev-change depending on type
Critical security fixes should be marked with kind/security

• kind/admin-change
• kind/dev-change
• kind/security
• [kind/adr](set-me)

What does this PR do / why do we need this PR?

...

• Fixes #

Information to reviewers

Checklist

• Proper commit message prefix on all commits
• Change checks:
  • The change is transparent
  • The change is disruptive
  • The change requires no migration steps
  • The change requires migration steps
• Documentation checks:
  • The public documentation required no updates
  • The public documentation required an update - [link to change](set-me)
• Metrics checks:
  • The metrics are still exposed and present in Grafana after the change
  • The metrics names didn't change (Grafana dashboards and Prometheus alerts required no updates)
  • The metrics names did change (Grafana dashboards and Prometheus alerts required an update)
• Logs checks:
  • The logs do not show any errors after the change
• PodSecurityPolicy checks:
  • Any changed Pod is covered by Kubernetes Pod Security Standards
  • Any changed Pod is covered by Gatekeeper Pod Security Policies
  • The change does not cause any Pods to be blocked by Pod Security Standards or Policies
• NetworkPolicy checks:
  • Any changed Pod is covered by Network Policies
  • The change does not cause any dropped packets in the NetworkPolicy Dashboard
• Audit checks:
  • The change does not cause any unnecessary Kubernetes audit events
  • The change requires changes to Kubernetes audit policy
• Falco checks:
  • The change does not cause any alerts to be generated by Falco
• Bug checks:
  • The bug fix is covered by regression tests

[rarescosma]

Have you checked what happens on existing clusters that are migrated to the latest Kubespray release?

This is OK to do if we can guarantee that this bit makes it into the cluster configuration:

(within ck8s-k8s-cluster.yaml)

post_kubeadm_hooks:
  - "{{ playbook_dir }}/../../playbooks/early_crds.yml"

[Mlundm]

@rarescosma
Should be okay since apps would already be installed on existing clusters but I will test it out for some practice with Kubespray.

[rarescosma]

How are the tests going for this? If all good, let's go ahead and merge -> made this a dependency for the cilium migration PR so we have less re-configuration to do on that side.

[Mlundm]

I would say its all good to merge.

Since there was no kubespray release to test with (cilium support only on latest), I ended up just checking how it would execute during upgrade_cluster.yml vs install.

Seems like post kubeadm hook is ran before cilium during install but not during upgrade. This should not cause any problems as far as I can see since servicemonitor CRDs should stay. If they get removed then we will have other problems such as needing to reapply all other servicemonitors after an kubespray upgrade.

[rarescosma]

Push the button? :)