Multi-level ownership for controller sharding

[Irori235]

Hi, I’m Hiroto (@Irori235) , on the same team as @hhiroshell. We’re very interested in controller sharding, and within our team we’ve been discussing a potential adoption blocker: multi-level ownership.

We noticed there are tasks on the project board related to multi-level ownership. We’ve drafted a few ideas, but if you already have a potential approach or direction, we’d love to hear it and collaborate on this.

What would you like to be added:
Multi-level ownership

Why is this needed:
We need to run a custom controller that implements multi-level ownership

Thanks!

[timebertt]

Happy to hear you're willing to contribute!
Can you outline your use case and your design ideas?

[Irori235]

Hi @timebertt, thanks for your response!
Let me share more details about our use case and the ideas we have been considering:

In our environment, we operate a large custom controller that converts abstracted custom resources for developers into concrete kubernetes resources. These resources often involve ownership hierarchies deeper than three generations. At current implementation, parent and child resources are located in the same shard by ownership references. However, this approach does not work when ownership hierarchies span more than three generations.

To address this, we are exploring a namespace-based sharding strategy. More specifically, we could use the namespace as the hash key when assigning a shard. Of course, we recognize that using the namespace as the hash key does not fit all use cases (e.g. when there is a significant imbalance in the number of resources across namespaces, shards will not be evenly assigned). This makes us wonder how we should design.

Another option we explored is to assign labels only on the namespace resource and have the controller check the namespace's label during every reconcile of the target resources. While this feels cleaner in theory, the controller-runtime does not support dynamically adjusting which namespaces of objects are cached. As a result, the controller would end up caching resources across all namespaces, which in our workload would create prohibitive overhead. Because we do not have a good solution for this issue, this approach seems less promising.

We would love to hear your thoughts on these ideas and whether you see other possible directions for handling multi-level ownership in sharding.

Thanks!

[timebertt]

I haven't had the time to design a mechanism for multi-level ownership yet, but I recognize the need for it.

When considering the typical 2-level ownership relationship (Deployment -> ReplicaSet -> Pod), where owned object names get another suffix appended, we could apply a prefix-based hash key. I.e., when hashing a Pod, strip the last two dash-separated segments; for a ReplicaSet, strip the last segment. With this, all objects would be assigned to the same shard as the top-level owner. This would also work for relationships with more than 2 levels, but requires compliant names.

Another idea would be to allow client-specified hash keys. I.e., if an object has the key.alpha.sharding.timebertt.dev/<controllerring-name> label/annotation, use its value as the hash key instead of a group/kind/namespace/name combination. As long as two objects (might be related by ownership, or not) have the same hash key, they are assigned to the same shard.
This allows flexible adaptation of the partitioning mechanism to the specific use case without complicating the generic sharding implementation. Clients would need to ensure enough variance in the used hash keys for a fair distribution of objects to shards.

Without going into more details, what are your thoughts on these ideas? Would they address your requirements?

[yuanying]

Thanks for the suggestion! I agree that using the key.alpha.sharding.timebertt.dev/<controllerring-name> label/annotation should work well for most cases.

However, in our setup, we need to manage objects that span across multiple clusters within a single shard. In that case, maybe this label-based approach doesn’t seem to be sufficient for us.

We’re therefore planning to proceed with the namespace-level sharding approach discussed in #680.