Can Kamaji read CRDs from a k3s API server and deploy workload cluster control plane pods on the host cluster?

[TheNilesh]

I’m testing an architecture involving Kamaji and k3s, and I’d like to confirm if the following setup is possible.

• There’s a k3s cluster acting as a virtual control plane (API server) for CRDs only
• The Kamaji CRDs (like TenantControlPlane) are created inside this k3s API server.

I want Kamaji to read these CRDs from the k3s API server but schedule the actual workload cluster control plane pods on a different cluster, let’s call it the host or management cluster.

The reason for this setup is that the k3s virtual control plane doesn’t support scheduling pods, so I want Kamaji to use it only for API/CRD management, while using the host cluster for the actual control plane components.

Is this kind of architecture supported by Kamaji?
If not, are there any recommended ways to separate the CRD source and the control plane pod deployment target?

[prometherion]

It's not supported, and honestly not getting the point of a such scenario.

You can use Vcluster or K3k since they seem addressing the scope.

[TheNilesh]

Point of this is to isolate CRDs and Controllers across customers. Each customer has multiple tenants. Thanks for suggesting vCluster.

[prometherion]

Kamaji aims for true isolation, such as the "sharing nothing principle" by leveraging on each customer bringing their own nodes.

[TheNilesh]

In our model, a customer is a top-level entity. Each customer brings their own HCP provider, CAPI core controllers and CRDs, but all of them still need to operate on the same underlying cluster. To make this possible, Kamaji must be able to read TenantControlPlane resources from one “source” cluster while creating the actual workload-cluster control plane deployments in a different cluster.

The k3s instance runs without a scheduler(in our case), and if Kamaji can use it purely as the API server for reading CRDs while relying on the host cluster to create the actual control-plane deployments, the system becomes far less shared. Each customer ends up operating inside their own isolated environment.

[prometherion]

If I understand correctly, all the instances should be controlled from a single entry point/single pane of glass, but being able to create Control Plane instances in different infrastructures.

This resonates in me the External Cluster Reference we designed for the Kamaji Control Plane provider for the CAPI ecosystem.

Tenants can define their desired clusters as CAPI resources, but they define in which Cluster the Tenant Control Plane instances (thus, the Control Plane pods) must be created.

If this is the case, we're supporting that using Cluster API: it's a validated design and already in production, since CLASTIX enterprise offering is based on this concept.