Test-Automation for rcm installation

[voigt]

We want to automate test runs to ensure the reliable installation of RCM and shims.

What to test

• test installation of RCM Helm Chart
• test successful installation of common shims (to be defined; I propose to start with the spin shim and expand from there)

Where to test

As RCM is messing with nodes' filesystem, using it on as many Kubernetes flavors as possible is crucial. Therefore, a minimum of covered Kubernetes distributions would be:

• kind
• k3s
• k3d

Ideally, we'd like to test the managed services of major cloud providers:

• AWS EKS
• Azure AKS
• GCP GKE
• ... potentially others like Civo, Digital Ocean, etc.

However, the main issue will be the associated cost of creating clusters during CI runtime. Maybe we'll find a way to get some sponsoring?

[phyrog]

Some pointers to useful GH actions:

• https://github.com/marketplace/actions/absaoss-k3d-action
• https://github.com/marketplace/actions/kind-cluster
• https://github.com/marketplace/actions/helm-chart-testing

[voigt]

security-profiles-operator faces similar challenges as rcm. Here their GH workflow to test within kind: https://github.com/kubernetes-sigs/security-profiles-operator/blob/main/.github/workflows/olm_tests.yaml

[phyrog]

Can we close this issue with the merge of #234?

[vdice]

I'd vote we either close this one and track next test automation goals (other distros, other shims, more scenarios) in a new ticket or continue tracking in this ticket but remove from the first milestone (I think #234 satisfies test automation needs for the first mvp release).

[vdice]

@voigt what are your thoughts on closing this initial issue now that #265 is in and covers all of the easy-to-CI K8s distros with the current supported shim (Spin)?

Then we can create issue(s) to cover adding some/all of the cloud vendor distros you've mentioned above... similarly added automation when other shims are supported?

[voigt]

I agree, lets close this issue once the open CI PRs are merged.

At this point, it is not clear whether we will ever be able to bring cloud vendor distros to CI (simply because of cost reasons). I'm unaware of another way to test apart from spinning up a (paid) cluster.