Add cilium support in Kubespray

[rarescosma]

PoC that Cilium is a viable CNI plugin for Kubespray, but boy there were a lot of hoops to jump through:

• On the base of our current release (2.27) cilium is still being installed using "homegrown" Kubespray templates. Upstream has changed this to use cilium-cli for installation, which internally uses helm which means we finally get access to a values.yaml template which we can modify. This is included in the v2.28.0 upstream release.
• A couple of fixes were cherry-picked to avoid connectivity issues in clusters that have installed cilium using the old method vs. the cli+helm-powered method.
  - Added extra values sections to enable the "policy audit mode" (because our built-in network policies were blocking traffic to the Kube API server and all hell broke loose - need further investigation on why this happens on apps apply).
  - Added extra values section to enable ServiceMonitors for the cilium agent pods and operator pods so the pretty Grafana dashboards added by Haorui are now showing pretty graphs.
• The values overrides are now handled in the parent repo

What type of PR is this?

Uncomment only one /kind <> line, hit enter to put that in a new line, and remove leading whitespaces from that line:

/kind api-change
/kind bug
/kind cleanup
/kind design
/kind documentation
/kind failing-test
/kind feature
/kind flake

What this PR does / why we need it:

Which issue(s) this PR fixes:

Fixes https://github.com/elastisys/ck8s-issue-tracker/issues/535

Special notes for your reviewer:

Worth knowing that the first few commits in the PR are cherry-picks, so not really in scope for change requests.

The only deviation from upstream (with high chances of getting upstreamed) is in this commit: 1d70e40

Also, might be worth reviewing in tandem with: elastisys/compliantkubernetes-kubespray#445

Does this PR introduce a user-facing change?:

[Xartos]

Question: Did you use master as the base or did you use the release-2.27.0-ck8s as base? Because that's the one that we are using for the 2.27 release

[rarescosma]

Question: Did you use master as the base or did you use the release-2.27.0-ck8s as base? Because that's the one that we are using for the 2.27 release

I used release-2.27.0-ck8s initially, then switched to master.

[Xartos]

Question: Did you use master as the base or did you use the release-2.27.0-ck8s as base? Because that's the one that we are using for the 2.27 release

I used release-2.27.0-ck8s initially, then switched to master.

You should probably switch to the release branch in that case since that's where this will be used

[rarescosma]

You should probably switch to the release branch in that case since that's where this will be used

Switched back to the release-2.27.0-ck8s branch as base.

[rarescosma]

Opened kubernetes-sigs#12408 for upstreaming the generic hooks required for this task.

[rarescosma]

Opened kubernetes-sigs#12408 for upstreaming the generic hooks required for this task.

Apparently there is another PR in the upstream pipes that will allow for generic helm values for cilium: kubernetes-sigs#12375 so we can take advantage of that instead of using hooks.

[rarescosma]

Closing in favor of #30 which consists entirely of cherry-picks and backports.