How to test a controller which uses several clusters?

[guettli]

Up to now we use our custom way to reconcile in several clusters.

One paint-point we have:

How to test that my controller reconciles correctly in several clusters?

Is testing on your roadmap?

[everpeace]

I could say envtest is powerful enough to test multicluster controller in multipole clusters.

Although it hasn't been merged yet, you can see multi cluster controller test example here:

• providers/cluster-inventory-api/suite_test.go: create two clusters hub and member
• providers/cluster-inventory-api/provider_test.go: test for multicluster configmap controller

[embik]

Hey @guettli, perfectly fine to discuss this in an issue IMHO. You can also reach out on #sig-multicluster (Kubernetes Slack).

As @everpeace said, I think the way to go is to use envtest. I could see us adding some wrappers around that in this repository to help with spinning up a local and multiple target clusters (this part depends on what provider you plug into multicluster-runtime though; e.g. for the kcp provider, we have our own envtest that spins up kcp instances).

If there is interest, I think it makes sense to work on this together as part of this repository. @everpeace's contribution is a great starting point for this. Can you maybe expand a little bit on what your pain points with testing controllers are? And how are you hoping multicluster-runtime could help with them?

[guettli]

@embik My current pain point about testing controllers which connect to several clusters: There is no good documentation. Somehow I can solve that, and get something working. But good documentation would be great.

I guess for most cases it does not matter how these clusters get created. It would be great if multicluster-runtime could provide docs and tools to test with several clusters. I think using kind-clusters would be good, because only few people are experienced with kcp.

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle stale
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[corentone]

This is a very interesting question and I hope we can share patterns!

To pinpoint what you mean by test, what would you expect be the level of testing? Integration vs Unit (or something in between!)
If The goal is to test the Controller logic (my guess): watches (maybe?) and reconcile (for sure!). You could swap the provider of clusters for something fake/local. If you want to test the provider as well, then it becomes more complicated as you'll need real clusters I think?

[guettli]

Currently, we do not use multicluster-runtime yet. When we test our controller which acts in several workload-clusters, then we use envTest and use the same cluster as fake workload-cluster. This works, but feels a bit hacky.

I started this thread because I was curious how testing is done with multicluster-runtime. It would be great if testing would be one part of the docs of multicluster-runtime.

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle rotten
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

[corentone]

/remove-lifecycle rotten

[corentone]

@guettli I think the how-to-create-clusters part is unlikely to be solved in multicluster-runtime consistently given the number of methods and people who may want to leverage different providers.

Could you maybe share ideas on how a testing documentation would look like to be useful?

Relinking @everpeace 's pointers which moved:
https://github.com/kubernetes-sigs/multicluster-runtime/blob/main/providers/cluster-inventory-api/suite_test.go
https://github.com/kubernetes-sigs/multicluster-runtime/blob/main/providers/cluster-inventory-api/provider_test.go

[guettli]

@corentone

I think the how-to-create-clusters part is unlikely to be solved in multicluster-runtime consistently given the number of methods and people who may want to leverage different providers.

yes

Maybe it is enough to create a section in the docs, which basically links to https://github.com/kubernetes-sigs/multicluster-runtime/blob/main/providers/cluster-inventory-api/provider_test.go

[skitt]

/sig multicluster