Add Cilium support in Kubespray 2.27

[rarescosma]

Warning

This is a public repository, ensure not to disclose:

• personal data beyond what is necessary for interacting with this pull request, nor
• business confidential information, such as customer names.

What kind of PR is this?

Required: Mark one of the following that is applicable:

• kind/feature
• kind/improvement
• kind/deprecation
• kind/documentation
• kind/clean-up
• kind/bug
• kind/other

Optional: Mark one or more of the following that are applicable:

Important

Breaking changes should be marked kind/admin-change or kind/dev-change depending on type
Critical security fixes should be marked with kind/security

• kind/admin-change
• kind/dev-change
• kind/security
• [kind/adr](set-me)

Release notes

Cilium values are controlled via group_vars/all/k8s_cluster/ck8s-cilium.yaml and mimic the structure of their CAPI equivalent.

Platform Administrator notice

A migration or devbox templates will be required to copy over the new group_vars/all/k8s_cluster/ck8s-cilium.yaml file.

Additionally, to enable the installation the following changes are required in group_vars/all/k8s_cluster/ck8s-k8s-cluster.yaml:

kube_network_plugin: cilium
cilium_identity_allocation_mode: "crd"
cilium_version: "v1.17.5"

cilium_enable_hubble: true
cilium_hubble_install: true
cilium_hubble_tls_generate: true

# See https://github.com/kubernetes-sigs/kubespray/issues/12276
kube_owner: root

What does this PR do / why do we need this PR?

Adds experimental support for running Cilium as CNI in Kubespray clusters.

• Fixes https://github.com/elastisys/ck8s-issue-tracker/issues/535

Information to reviewers

Checklist

• Proper commit message prefix on all commits
• Change checks:
  • The change is transparent
  • The change is disruptive
  • The change requires no migration steps
  • The change requires migration steps
• Documentation checks:
  • The public documentation required no updates
  • The public documentation required an update - [link to change](set-me)
• Metrics checks:
  • The metrics are still exposed and present in Grafana after the change
  • The metrics names didn't change (Grafana dashboards and Prometheus alerts required no updates)
  • The metrics names did change (Grafana dashboards and Prometheus alerts required an update)
• Logs checks:
  • The logs do not show any errors after the change
• PodSecurityPolicy checks:
  • Any changed Pod is covered by Kubernetes Pod Security Standards
  • Any changed Pod is covered by Gatekeeper Pod Security Policies
  • The change does not cause any Pods to be blocked by Pod Security Standards or Policies
• NetworkPolicy checks:
  • Any changed Pod is covered by Network Policies
  • The change does not cause any dropped packets in the NetworkPolicy Dashboard
• Audit checks:
  • The change does not cause any unnecessary Kubernetes audit events
  • The change requires changes to Kubernetes audit policy
• Falco checks:
  • The change does not cause any alerts to be generated by Falco
• Bug checks:
  • The bug fix is covered by regression tests

[rarescosma]

Following the CAPI lead, the installation of ServiceMonitors is now disabled by default.

[rarescosma]

Dropping a reminder here that the "install ServiceMonitor CRD early" solution from CAPI needs to be ported to Kubespray as well.

[rarescosma]

Yes, I agree. It would be nice if we can be as close to upstream as possible, I think we've been lagging behind on getting things upstreamed recently but as a general rule it makes sense to only make changes that at least are possible to upstream.

Changed the implementation so that we only require backports and cherry-picks from upstream

@Ajarmar can you take another look when you've got some spare time?

[lucianvlad]

Nice work!

[rarescosma]

Referencing changes to base config in CAPI that should be mirrored here, once the Cilium initiative on Kubespray picks up steam again: https://github.com/elastisys/ck8s-cluster-api/pull/406

[Ajarmar]

LGTM! I like the new solution (haven't tested though).

I have one question: do we want any of this on main as well, for v2.28 and following releases?

[rarescosma]

LGTM! I like the new solution (haven't tested though).

I have one question: do we want any of this on main as well, for v2.28 and following releases?

Thank you!

Why would we not want it on main plus the 2.28?
Feels weird to introduce a feature just for an old release branch :)

Also, I'd like to mention that upstream has some interesting ongoing work for installing CRDs early into the clusters, which might let us remove some of the diff in our fork.