Skip to content

Create Position around Hub Cluster (prev. management Cluster) #8210

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 5 commits into
base: master
Choose a base branch
from
+57 −0
Open

Create Position around Hub Cluster (prev. management Cluster) #8210

Changes from 1 commit
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
f383444
Create management-cluster-position-statement.md
corentone Dec 9, 2024
90a99eb
Update and rename management-cluster-position-statement.md to hub-clu…
corentone Jan 28, 2025
d0fbe3d
Update sig-multicluster/hub-cluster-position-statement.md
corentone Feb 4, 2025
f5dc455
Update hub-cluster-position-statement.md
corentone Feb 18, 2025
74b2d90
Update hub-cluster-position-statement.md
corentone May 5, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
46 changes: 46 additions & 0 deletions sig-multicluster/management-cluster-position-statement.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,46 @@
# Management Cluster - SIG Multicluster Position Statement
Copy link
Member

@MikeSpreitzer MikeSpreitzer Dec 12, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not all multi-cluster management designs have a monolith. I do not think this SIG should take a position that requires or recommends a monolith. There are multiple roles involved in multi-cluster management. I think that it may be more helpful to identify those roles. A system with a monolith can be described as having one cluster that plays all those roles.

For example, in https://github.com/kubestellar/kubestellar/ we identify distinct roles and allow flexibility in what plays which role. We start by identifying a concept that is less than a "cluster". We define a "space" to be the fragment of cluster behavior that is only concerned with generic API machinery stuff. A space can store and serve kube API objects and subjects them to the general-purpose controllers (the ones that apply to all kinds of API objects, not the controllers involved specifically with containerized workload (Pod, Service, ...) ). KubeStellar defines the following roles.

  • an "inventory space" holds the inventory of managed clusters. That is, API objects that refer to or describe the managed clusters. In OCM these are ManagedCluster objects.
  • a "workload description space (WDS)" holds both (a) objects that describe the workload (both desired state from users and reported state to users) and (b) control objects that control the multi-cluster management behavior.
  • a "workload execution cluster (WEC)" is a managed cluster, where the workload is run or executed.
  • the current KubeStellar implementation uses OCM and has a "transport space" that holds the workload interface objects of OCM (e.g., ManifestWork). The current implementation of KubeStellar requires that one space play both "inventory space" and "transport space" roles, calling this joint role "inventory and transport space (ITS)".
  • the current implementation of KubeStellar uses https://github.com/kubestellar/kubeflex as a space manager. The KubeFlex API objects reside in, and the KubeFlex central controller runs in, a role called the "KubeFlex hosting cluster".
  • KubeFlex considers itself to manage "ControlPlanes", and supports a few different types. One is "vcluster", which uses https://www.vcluster.com/ to create virtual slices of its hosting cluster. Another is "k8s", which is essentially hosted Kubernetes kube-apiserver, its storage, and general-purpose controllers (the essence of a "space"). KubeFlex can also have a ControlPlane that represents its hosting cluster. We are working on enabling KubeFlex to adopt an arbitrary existing cluster as a ControlPlane.

One configuration that KubeStellar supports is one real cluster playing the roles of WDS, ITS, and KubeFlex hosting cluster.

In OCM, where the workload description is wrapped and the workload execution cluster holds the unwrapped objects, could one cluster play both WDS and WEC roles?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for those details. I agree with your statement of not being a monolith.
I'm only trying to define the nuance of what a management cluster is; I don't think I force everything to be in the same or a monolith. Maybe my wording is wrong? I could rename the title to be "Management Clusters" to make it clearer.... but I'm just trying to get a wide definition of what such a cluster is (by opposition to a workload cluster).

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wonder what is Kubestellar recommended way to host WDS/ITS/Kubeflex?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Q: If the Inventory Space and the Workload Description Space are truly separate, how can we define an API that allows to schedule a workload on the clusters in the inventory?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@MikeSpreitzer I think those are questions for you. Let me know if I can help.


Author: Corentin Debains (**[@corentone](https://github.com/corentone)**), Google
Last Edit: 2024/12/09
Status: DRAFT

## Goal
To establish a standard definition for a central cluster that is leveraged by multicluster
controllers to manage multicluster applications or features across an inventory of clusters.

## Context
Multicluster controllers have always needed a place to run. This may happen in external
proprietary control-planes but for more generic platforms, it has been natural for the
Kubernetes community to leverage a Kubernetes Cluster and the existing api-machinery
available. There has been a variety of examples of which we can quote ArgoCD, MultiKueue
or any of the Federation effort (Karmada, KubeAdmiral), all of them not-naming the "location"
where they run or not aligning on the name (Admin cluster, Hub Cluster, Manager Cluster...).
The [ClusterInventory](https://github.com/kubernetes/enhancements/blob/master/keps/sig-multicluster/4322-cluster-inventory/README.md)
(ClusterProfile CRDs) is also the starting point for a lot of multicluster controllers and,
being a CRD, it requires an api-machinery to host it.

## Definition

A (multicluster) management cluster is a Kubernetes cluster that acts as a
control-plane for other Kubernetes clusters (named Workload Clusters to differentiate
them). It MUST have visibility over the available clusters and MAY have administrative
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wonder what does "visibility" mean here? There can be workload clusters that do not have public IP.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed, visibility is vague. I think it is somewhat the ability to query more data about the cluster. I'm thinking it would be the ability to "READ" via kubectl (thought it may not be everything). But it should be able to potentially see the Cluster object in GKE or other platforms (wherever the provisioning/lifecycle happens).

Maybe this is too vague, I originally had this in to mandate some kind of visibility on the cluster so that the controller could do something (if the controller has only access to the clusterprofile, I dont know what it can really do).

But we may be able to remove it and it wouldn't affect the definition much.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think that MUST is correct then.
several existing multi-cluster systems work without the hub being able to read the apiserver of the member.
The reason is that having 1 system (the hub) able to read ALL members is a different security model than having each individual member able to access the hub.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@serbrech good point; the must is likely too strong then. Would a "should" be okay? Or should it be a "may"? Now, even if we reverse the security model (e.g. work api model where the cluster reaches out the central cluster), there is still power from the hub over the workload cluster given it.

Here is what I could change:

Suggested change
them). It MUST have visibility over the available clusters and MAY have administrative
them). It MUST have the ClusterProfiles written on it MAY have visibility over the available clusters and have administrative

(I could also drop the clusterprofile part as obvious)

Copy link
Member

@serbrech serbrech Jan 21, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for revisiting. I'm aligned with MAY.
Specific features could require this visibility, but not the general use case of a hub cluster.

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In my systems, the workload clusters pull from the management cluster. Controllers run on the workload clusters so MAY have administrative privileges works well in my world.

Visible is interesting for my world; my management clusters are authoritative for the data and the workload clusters reconcile against it. But, there MAY be two management clusters from which a workload cluster pulls (i.e. a dev and prod source-of-truth); each management cluster authoritative for particular data.

Lastly, could visibility be decomposed into data storage about a workload cluster and mechanism for transport/reconciliation between the hub/workload cluster?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've removed visibility to be "access to api, metrics or workloads". So it could be that it doesn't access any of those or a subset of them.

@cbaenziger I'm not sure what you mean about data storage for visibility, apiserver/etcd?

Visible is interesting for my world; my management clusters are authoritative for the data and the workload clusters reconcile against it. But, there MAY be two management clusters from which a workload cluster pulls (i.e. a dev and prod source-of-truth); each management cluster authoritative for particular data.

we should talk more about this example. There could be multiple multiple management clusters for a given cluster, and the definition allows for it.
Now putting my infra hat (not PR writer), given the current isolation of kubernetes between workloads, I would prefer to mix Business Units with a similar risk) instead of different environments (prod vs dev).

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think I agree with definition as written now. To clarify my operation:

Lastly, could visibility be decomposed into data storage about a workload cluster and mechanism for transport/reconciliation between the hub/workload cluster?

Data: My hub clusters store data in their etcd (kine based) about the workload clusters. However, they are not responsible for propagating that configuration data to the workload clusters.

Transport: The workload clusters run a controller which polls (an interposing service backed by the) hub cluster and synchronizes specific resources to the workload cluster. The hub cluster is not responsible for the propagation of the state it stores.

As to multiple hub clusters: Since workload clusters are responsible for synchronizing their resources from the hub clusters, one hub cluster could be responsible for storing a resource that drives namespace definitions. Another could be responsible for storing RBAC policy definitions. Or as mentioned, one could hold dev resources and prod resources -- I fully agree with your operator perspective that no workload cluster ought to sync from both a dev and prod system -- merely that each is authoritative for its tranche of data; see the namespace vs RBAC split for a less broken operational example.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@cbaenziger thank you for the details. Indeed your model is similar to the WorkAPI (where there is an agent running in the workload cluster pulling data from the hub cluster).
I agree with the statement on multiple clusters.

One thing I'd note is that even if you "pull" the data from the hub cluster, the hub cluster still holds permissions (although indirectly) on the remote clusters since it could technically go wild and "tell" the workload-cluster-pullers that they need to run compromised code. It wouldn't be able to DDOS the apiserver but if the puller pulled RBAC and not restricted to a few namespaces, it could escalate privileges easily.

privileges over them. It SHOULD not be part of workload clusters to provide a better
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am not convinced of this "SHOULD". I am not convinced that this sort of statement belongs in a definition.

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wonder if you are not convinced that the "hub" cluster and a "spoke" cluster should not be the same or just "should" does not belong to a "definition"? I am curious about the reason if it's the former.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@MikeSpreitzer could you clarify your statement please? If you mean that the management cluster relationship to workload clusters should be different than what the definition says, could you clarify your position (they MUST isolated, the definition shouldn't talk about their relationship or the SHOULD is too strong and it should be "Management cluster MAY also be a workload cluster")

security isolation, especially when it has any administrative privileges over them.
There MAY be multiple management clusters overseeing the same set of Workload Clusters
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This statement supposes multiplicity only in the form of potentially competing equals; it omits the possibility of clusters fulfilling distinct roles.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I tried to clarify in the next section that it could be multiple roles:

to allow for separation of functionality (security-enforcer management cluster vs
config-delivery management cluster)

is the wording not strong enough? I don't mean to close that door and thought the current wording was enough and not emitting a direction, just requiring that the admin oversees potential overlap between different management clusters. If there is no overlap, they are fine to co-exist as separate clusters.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@MikeSpreitzer Did that answer your comment (e.g. can I resolve). I'm not sure I fully understood it and tried my best to answer it. Please let me know if I missed it.

and it is left to the administrator to guarantee that they don't compete in their
management tasks. There SHOULD be a single clusterset managed by a management cluster.
Management clusters can be used for both control-plane or data-plane features.
Copy link
Member

@MikeSpreitzer MikeSpreitzer Jan 22, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it would be helpful to say what is meant here by "control-plane" and "data-plane".

In the call on Tuesday I heard something like the data plane is about the workload and the control plane is about controlling the workload's propagation from hub to execution clusters. (I specifically say "execution" clusters to contrast with where the authoritative description of the workload lives in the hub, which might be in a cluster or something like it. In KubeStellar we keep the hub-side authoritative description of the workload un-wrapped in what we call a "space", which something that has (at least) a cluster's API machinery.)

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I tried to clarify by differentiating "Business applications" with "Infrastructure".
I steered away from execution because people think about infra pods which are being run and get confused on where they would go. I think kubestellar "execution cluster" is equal to what I'd call "workload cluster".
I think your "space" would live on a hub cluster, are you suggesting we should separate the definition of hub and space cluster? (separating infra and app-infra)

Side notes that I may add to the definition/doc if that helps and the current wording doesn't cover it:

There are two types of Workloads: Business/Product/application vs Infrastructure/Platform.
There are two personas and the configs they manage: Platform/Infra Team and Product/Application/Business teams.

I consider workload clusters as running (e.g. pods serving traffic or achieving functionality) business applications. Workload clusters may also run infrastructure applications that assist them in serving.
Hub clusters run infrastructure applications and may hold infrastructure/definitions for business applications.



### Rationale on some specific points of the definition

* Multiple management clusters: While it often makes sense to have a single "Brain" overseeing
a Fleet of Clusters, there is a need for flexibility over the number of management clusters. To
allow redundancy to improve reliability, to allow sharding of responsibility (for regionalized
controllers), to allow for separation of functionality (security-enforcer management cluster vs
config-delivery management cluster), to allow for migrations (from old management cluster to new
management cluster) and likely more.
* Management cluster also being part of the workload-running Fleet: We do recommend that the
management cluster(s) be isolated from the running Workload Fleet for security and management
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This makes sense in general, but I am not convinced that there are no use cases for combining roles in one cluster.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the first paragraph encourages different clusters for different roles actually.
This second paragraph is just about being part of the workload clusters or not.

Let me try to think of introducing the notion of "role" or something like that, as a subdivision of the broad Management Cluster.

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wonder what's the definition of "workload"? I usually associate them with applications but not controllers so it's okey to me to run controllers in the "central" cluster that requires leader-election.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it is definitely okay to run a controller in the management cluster. A controller doesn't have to be a "Workload" and be considered part of the "control-plane".

I think its the persona that matters. If it is a platform admin -owned controller, performing management tasks, I wouldn't consider it a workload. Workload to me is an application serving actual business-logic purpose.

All in all, running those management controllers is the reason why I want to define management clusters and not just a management API. (we had discussed internally giving simply an API with machinery... but then very quickly you want to bring a controller to act on this API and look where to run it)

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@ryanzhang-oss should I define workload more? I think a sentence in this doc may be enough.

Workload works great when opposing management, for hub maybe we need to say spoke? or is workload still good?

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AFAIK, workload in k8s basically means things like deployment/daemonset/statefulset, so it can be anything. I am not sure if there is a way to say the deployment contains a controller vs a real application.

concerns. But there may be specific cases or applications that require to mix the two. For example,
controllers that take a "leader-election" approach and want a smaller footprint.