Support for sharded controller instances

[embik]

This has been on my mind for a while and I wanted to create an issue for the backlog.

Problem Statement

In the main controller-runtime world, you usually run one controller instance per Kubernetes cluster. That results in some "natural" limit to scaling considerations since it's a single Kubernetes cluster. There are some solutions in the community to build sharded controllers.

multicluster-runtime has an obvious challenge, and that is a huge amount of clusters creating a lot of load on the controller instance. Vertical scaling (increasing CPU and memory) will get you only so far. The obvious sharding boundary are the cluster.Cluster engaged by multicluster-runtime.

Feature Description

It would be ideal to solve this problem at the multicluster-runtime level. I can see providers implementing options (e.g. label selectors) that help with manual or automated sharding specific to the provider (e.g. not all cluster fleets will be based on CRDs and such, a label selector won't work), but a generic solution you automatically get would be ideal and a real "selling point" for multicluster-runtime.

a mcmanager.Manager instance could be configured to shard by cluster names, maybe creating some sort of hash ring to decide which manager instance is responsible for a particular cluster coming from the provider. There is also the possibility to require this from providers somehow, depending on whether we need information or awareness from providers, and have a secondary ShardedProvider interface that provider implementations can optionally implement in addition to the Provider one.

[FourFifthsCode]

This sounds like a really great feature and I think you're right, we'll probably hit scaling issues sooner than later when going multi-cluster.

[Colvin-Y]

This is a really useful user case, maybe we can refer to
Controller Sharding | KubeVela
https://github.com/kubevela/kubevela/blob/master/design/vela-core/sharding.md
Feat: support sharding in controller (#5360) · kubevela/kubevela@9efbb72 (github.com)

[ntnn]

My first idea was to have providers supply clusters with labels (or something like it). But I think that wouldn't really solve the problem.
Providers (all providers) would have to guess which data is useful for eventual users to use the labels to separate their manager instances. Not to mention for things like the kubeconfig provider, which is reading kubeconfigs from secrets, there is no context available. At most it could read the labels of the secret.
But that seems impractical.

On top - I think that for a lot of users the logic would be environment-dependent, so it would make more sense to leave the implementation details of the selection to the user.

I think it would make more sense for the manager to have a filter function, so when a provider engages a cluster with the manager the manager runs the cluster through the filter and decides based on that return value if it should handle the cluster.

That would probably go into the mcmanager.Options.

E.g. for region-based, but this could be anything:

myRegion := "emea" // e.g. read from the local cluster node

provider, err := myprovider.New(myprovider.Options{})

mgr, err := mcmanager.New(ctrl.GetConfigOrdie(), provider, mcmanager.Options{
	LeaderElectionID: myRegion + ".1234.example.com",
	FilterCluster: func(ctx context.Cotnext, cl cluster.Cluster) (bool, error) {
		client := cl.GetClient()

		var nodes corev1.NodeList
		if err := c.List(ctx, &nodes); err != nil {
			return err
		}

		if len(nodes.Items) == 0 {
			return fmt.Errorf("could not read cluster nodes")
		}

		for _, node := range nodes.Items {
			clusterRegion, ok := node.Labels["topology.kubernetes.io/region"]
			if !ok {
				continue
			}
			if clusterRegion == myRegion {
				return true, nil // region matches, this manager should handle this cluster
			}
		}
		return false // region does not match, manager does not engage the cluster 
	},
})

return mgr.Run(ctx)

Changing .Engage appropriately:

func (m *mcManager) Engage(ctx context.Context, name string, cl cluster.Cluster) error {
	if m.opts.FilterCluster != nil {
		engage, err := m.opts.FilterCluster(ctx, cl)
		if err != nil {
			return err
		}
		if !engage {
			return nil
		}
	}
	ctx, cancel := context.WithCancel(ctx) //nolint:govet // cancel is called in the error case only.
	for _, r := range m.mcRunnables {
		if err := r.Engage(ctx, name, cl); err != nil {
			cancel()
			return fmt.Errorf("failed to engage cluster %q: %w", name, err)
		}
	}
	return nil //nolint:govet // cancel is called in the error case only.
}

That would need a test drive, but it would require no changes for providers, no change for users who do not need a multi-controller environment, make it easy to plug in if a multi-controller environment is later needed and it would be flexible enough to allow users to implement whatever logic they need.

A sensible default could maybe be something using leases in the local cluster to enable the automatic "separation of duty"? Or creating the lease in the target cluster, so the controller can run in different clusters.

Also on mcmanager.Options:

multicluster-runtime/pkg/manager/manager.go

Line 115 in a41032c

type Options = manager.Options

I don't think this Options is actually used. Everything references manager.Options, but I think the intent was to alias manager.Options and to later change the alias to a multicluster Options? And it's probably an alias now because no additional fields are needed and CI probably doesn't like a struct with a single embed without anything more.
That works now with users passing in mcmanager.Options but the references in the rest of the file should be updated.

[embik]

A filter would be something worth exploring, but ideally I would like to see even more functionality baked into multicluster-runtime and account for dynamic situations too. It would be really neat to have a consistent hash ring built in so that controller instances dynamically pick up or drop clusters based on how many peers are around.

[corentone]

This is a very interesting idea and I look forward to exploring more.
I do think Filter would get us a long way, but it would likely be a provider thing? (Unless the cluster.Cluster object has labels too?). We may even be able to leave it to providers for now (maybe different providers would have different ways?) and users to come up with ideas that fit their needs?

One of the main greatness for mcmanager is the ability to share states/actions across clusters so I'd want to have an opinion, suggestion or implementation on how it would look like between controllers.

[ntnn]

I do think Filter would get us a long way, but it would likely be a provider thing?

I could see a meta or middleware filter provider - like the multi provider - instead of putting the filter into the manager itself.
But it shouldn't be something the "raw" providers that actually gather and generate clusters should concern themselves with.

Unless the cluster.Cluster object has labels too?

It doesn't and since cluster.Cluster is coming from controller-runtime that would need to be changed there.

And I'm not sure if providers adding arbitrary labels would help, since they don't all have access to the same information on the clusters - mcr could offer a standard implementation that gathers some information from well known labels and annotations, but the success of that would also depend on having access to that information.
And it would depend on the providers utilizing that standard implementation to fill the labels.

If cluster.Cluster would start having labels there would certainly be feature requests coming up to select a set of clusters based on a label selector which the current interfaces and implementations do not support.
That wouldn't be bad but I still think providers settings labels wouldn't be the right idea, they should just focus on procuring the cluster information.

One of the main greatness for mcmanager is the ability to share states/actions across clusters

I agree - for that reason I'd prefer that the operators using mcmanager also aren't bound to run in a single cluster; which makes synchronization between them a bit hard.

It would be really neat to have a consistent hash ring built in

Maybe a bit of a mad idea - I had previously explored multi-controller environment by having a controller set an annotation on an object before processing it (and remove the annotation after processing).
How about mcr manager setting their reference on an annotation like a multicluster owner reference with an ID and timestamp?
If the ID doesn't match the manager it doesn't process the object.
Periodically the timestamp is updated and if it wasn't updated in a longer timeframe another mcmanager just takes over?

[zachsmith1]

Hey folks 👋 — excited to share a WIP that adds initial sharding for multi-cluster controllers without any upstream changes.

Uses Rendezvous (HRW) hashing over the live peer set to choose an owner per downstream “cluster” (namespace if using ns provider).

Enforces single-writer with per-cluster fencing leases; owners cleanly detach/re-attach watches on handoff.

Auto-rebalances when you scale replicas up/down (or when peers join/leave).

This is a minimal, working baseline to start the discussion. Happy to iterate on the API/ergonomics. Future directions I’d love feedback on:

Geo-aware/weighted placement

Policy layer (taints/tolerations, labels/affinity) to constrain/steer which controllers manage which clusters

Open to suggestions and questions!

Please give it a shot here: https://github.com/zachsmith1/multicluster-runtime/pull/1/files

[ntnn]

I've had a quick look and that's pretty much what I had in mind with using leases.

The client used for lease management and peering should be configurable, but default to the local cluster.
Though that'd need the mcmanager.Options to be usable to set options for mcr #64

That would make two things possible:

1. MCR-based controllers from different clusters can synchronize by configuring the same cluster client to manage leases in
2. Geo-aware placement can be managed by using clients for different clusters based on the location and filtering providers to only accept clusters in that area.

For weighed placement, taints, tolerations and affinity the major problem would be getting the information. I don't think clusters expose that information to all clients and I don't think it should be put on the providers' back.

Maybe it'd make sense to start a multicluster.Cluster that supports setting and getting labels and annotations. That would allow providers to fill if known (like in cluster-api), users can add their own provider middlewares to add labels and annotations and that in turn would enable things like affinity to be handled in the manager.

[FourFifthsCode]

Maybe it'd make sense to start a multicluster.Cluster that supports setting and getting labels and annotations. That would allow providers to fill if known (like in cluster-api), users can add their own provider middlewares to add labels and annotations and that in turn would enable things like affinity to be handled in the manager.

A middleware approach to the Cluster struct sounds very interesting and pluggable! 🚀

[timebertt]

Have you seen https://github.com/timebertt/kubernetes-controller-sharding?

[embik]

I have, it partially inspired this issue!

[FourFifthsCode]

Another thing that would be really nice to think about is not just sharding at the cluster level but also spreading some of the load across instances for large clusters too if possible. Maybe that's already being considered but if not, would be nice to look at too.

[zachsmith1]

Opened up this PR here: #74