Multiple CAPI controllers on one CAPI management Cluster with Namespace Selector

[jiangfwa]

What would you like to be added (User Story)?

As a operator, I would like to be able to install multiple CAPI controllers on one CAPI management cluster. Each CAPI controllers have its namespace selector to only reconcile the clusters under the selected namespaces.

Detailed Description

In the production, if there are more than 10+ openstack clusters, each cluster has around 1000+ nodes. The capi-controller and capo-controller are under heavy pressure.

In this case, my idea is to have multiple sets of capi-controllers, and sort the app clusters into multiple sets. Let each set of capi-controllers only manage its app clusters. For example, I can put the label on each Cluster resource and the namespace as following:

- namespace `cluster-a1` with label `env: alpha`
- namespace `cluster-a2` with label `env: alpha`
- namespace `cluster-b1` with label `env: beta`
- namespace `cluster-b2` with label `env: beta`
- namespace `cluster-p1` with label `env: prod`
- namespace `cluster-p2` with label `env: prod`

Then, if multiple sets of capi-controllers installed, and with the namespace selector, each capi-controller can work as the following:

- namespace: `capi-mgmnt-alpha`
    - capi-controller with namespace selector `env: alpha`
    - capa-controller with namespace selector `env: alpha`
    - capo-controller with namespace selector `env: alpha`
- namespace: `capi-mgmnt-beta`
    - capi-controller with namespace selector `env: beta`
    - capa-controller with namespace selector `env: beta`
    - capo-controller with namespace selector `env: beta`
- namespace: `capi-mgmnt-prod`
    - capi-controller with namespace selector `env: prod`
    - capa-controller with namespace selector `env: prod`
    - capo-controller with namespace selector `env: prod`

This can do some kind of load balancing on the reconciling of these clusters to multiple capi-controllers. For the CRD, all sets of the capi-controllers need to share the CRD, so the versions between these capi-controllers need to be the same or at least support the same version of CRDs.

Anything else you would like to add?

N/A

Label(s) to be applied

/kind feature
One or more /area label. See https://github.com/kubernetes-sigs/cluster-api/labels?q=area for the list of labels.

[k8s-ci-robot]

This issue is currently awaiting triage.

If CAPI contributors determine this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[fabriziopandini]

As a project we already tried the path of running multiple instance of controllers, and we faced many so many operational issues that we decided to revert course. (some notes https://cluster-api.sigs.k8s.io/developer/core/support-multiple-instances).

I don't think we want or we should reconsider this idea, also because things got even more complex over time, and now we will have to deal with even more sophisticated caching, cluster wide resources, cross namespace dependencies across resources and probably even more.

If the problem if scale, and you are not willing to break down your management cluster into a few management clusters, I think the simplest and more viable way to follow is to improve what a cluster API instance can manage.

Just fyi, this is what we as maintainer we are working toward, e.g. by working with the controller-runtime team to develop thinks like priority queue, to get rid of the blockers we faced in the past when working at scale.

If you can provide more info about the issue you are facing at scale, might be that we can guide you to make progress because some options already exist in CAPI, but they are not active by default