Comparison with Karmada?

[vicaya]

The architectures are similar with a hub/member design and declarative scheduling CRs. What are the pain points that are only addressed by kubefleet?

My current top use case is not well addressed by either so far: application failover AND recovery with ordered cluster affinities:

1. Application and its dependent resources have cluster affinities in a preference order.
2. Application and its dependent resources fails over according to the order.
3. Application and its dependent resources move back to the highest order cluster when it recovers.

Karmada can already address one-time failover reasonably well. There is an open item to address this. I wonder if kubefleet can already address this use case?

[vicaya]

A cursory look seem to indicate that kubefleet cannot address this yet, because all the cluster affinity types are *IgnoredDuringExecution. It'd be great we have a "alwaysPreferred" scheduling type that can trigger a rescheduling when an expression of (custom) metrics (a la hpa/vpa) returns true.

Karmada has a WorkloadRebalancer CR that might eventually work for this.

[vicaya]

High level product requirements for this feature:

1. Minimize regional latency in a fault tolerant manner: regional workload is scheduled to a primary regional cluster for best latency. The workload should fail over to a backup cluster in another region in case of DC outages, with higher latency. The workload should move back to the primary cluster when it recovers for optimal latency.
2. Minimize cost in cloud burst: workload is scheduled to an on-prem cluster until its fully utilized. More workloads should be scheduled to a cloud cluster when the on-prem cluster is full. Workloads should move back to the on-prem cluster when its utilization dips under a threshold to save cloud cost.