[BUG] Upgrade Failures: Lack of Persistent Webhook and Multi-Replica Support

[Divya1388]

Summary

We are self-hosting Kubefleet and have encountered critical issues impacting pod
creation and upgrade reliability. These issues hinder our ability to meet
operational requirements as a regulated industry. Below, we outline the challenges
and propose potential solutions.

Issues and Observations

1. Pod Creation Blocked Outside Fleet-* and Kube-* Namespaces

• The ValidatingWebhookConfiguration blocks the creation of pods in namespaces
  other than fleet-* and kube-*.

• As a regulated industry, we require agents and other critical workloads to run in
  custom namespaces, but the webhook prevents this.

• Manual patching of the webhook (e.g., setting failurePolicy: Ignore) is required
  to unblock pod creation, which is not sustainable.

  As a workaround, we have resorted to creating more namespaces with fleet-* and
  kube-* prefixes. However, this approach is not scalable and introduces
  operational complexity. For instance, CustomResourcePlacement (CRP) objects fail
  to be created in the hub namespace (e.g., hub-namespace) but are successfully
  created in member namespaces (e.g., member-namespace-1).

  Alternatively, if we use a fleet-* namespace, the CRP throws the following error:

  • 

  This inconsistency leads to:

  • Increased namespace sprawl, making cluster management more challenging.
  • Potential conflicts with existing naming conventions and policies.
  • Difficulty in maintaining compliance with regulatory requirements.

  What do you propose as a sustainable solution to address this limitation, and what
  are the potential implications of continuing with this workaround?

2. Upgrade Failures for Node Image and Kubernetes Upgrades

• During node image and Kubernetes upgrades, we observe failures due to the
  inability of the hub-agent or other pods to come up because of the
  ValidatingWebhookConfiguration.

• The two alternatives we tried are as follows:

  1. Webhook Configuration:

     • The ValidatingWebhookConfiguration must be manually patched after every
       upgrade or node pool eviction to unblock pod creation.

     • Manual changes are lost because the controller deletes and recreates the webhook
       configuration using a delete-and-create pattern:

       if err := w.mgr.GetClient().Create(ctx, &validatingWebhookConfig); err != nil {
             if !apierrors.IsAlreadyExists(err) {
                   return err
             }
             // Overwrite: delete and recreate
             if err := w.mgr.GetClient().Delete(ctx, &validatingWebhookConfig); err != nil {
                   return err;
             }
             if err = w.mgr.GetClient().Create(ctx, &validatingWebhookConfig); err != nil {
                   return err;
             }
       }

  2. Multi-Replica Hub-Agent:

     • To improve high availability, we added a rolling update strategy, set replicas
       to 5, and configured a PodDisruptionBudget.

     • Certificates are hardcoded in the webhook server setup, as seen in
       cmd/hubagent/main.go,
       where FleetWebhookCertDir and FleetWebhookPort are used without dynamic
       synchronization.

     • Each replica must use the same TLS certificate stored in FleetWebhookCertDir.
       If the certificates are not synchronized across replicas, the API server might
       receive different certificates from different replicas, leading to TLS handshake
       failures.

     • When multiple replicas are behind a Kubernetes Service, the API server's
       requests are load-balanced across the replicas. If the replicas do not share the
       same certificate, the API server might encounter inconsistent certificates
       during successive requests.

     • The API server logs errors like:

       x509: certificate signed by unknown authority
       

       This disrupts the admission process, potentially causing delays or failures in
       pod creation.

     • The caBundle field in the ValidatingWebhookConfiguration is critical for
       ensuring the API server trusts the webhook service. For more details, refer to
       the Kubernetes documentation on
       Admission Webhooks.

Proposed Solutions

1. Persistent Webhook Configuration:

   • Support persistent configuration for webhook fields (e.g., failurePolicy, ) to
     avoid manual patching.
   • Can other namespaces be ignored to simplify webhook configuration and reduce
     operational overhead?

2. Multi-Replica Hub-Agent:

   • Address TLS handshake errors by:
     • Using a shared certificate source (e.g., Kubernetes secrets) to prevent
       mismatches.
   • Wire the replicas and strategy fields into the hub-agent deployment and add
     PodDisruptionBudgets for upgrade reliability.

3. Namespace Flexibility:

   • Allow pod creation in custom namespaces by updating the webhook configuration to
     support broader namespace scopes.

Request and Questions

• Can you assist in resolving the TLS issues with multiple replicas to ensure
  upgrades do not fail?
• Would you be open to supporting persistent webhook configuration and namespace
  flexibility?

[ryanzhang-oss]

@Divya1388 , Thank you very much for raising the issue. I wonder if you actually used azure/fleet or kubefleet (https://github.com/kubefleet-dev/kubefleet) which is the CNCF project. This repo is served as the backend for the azure kubernete fleet manager (https://docs.azure.cn/en-us/kubernetes-fleet/). They are close but not entirely the same.

1. Pod Creation Blocked Outside Fleet-* and Kube-* Namespaces

This is by design, the hub cluster is not designed to run workload. I wonder if you have any reason to run the pod in the hub cluster?

I wonder if you can describe to us your use cases?

Upgrade Failures for Node Image and Kubernetes Upgrades

We will look into this and get back to you.

[tillig]

Possibly related: kubefleet-dev/kubefleet#290

@Divya1388 is on the same team as me. We are using the CNCF version to self-host. We recognize that this issue has perhaps been filed in the wrong repo. Sorry, it gets confusing.

Use cases are outlined in the other issue. Things like:

• Prometheus
• OpenTelemetry
• Security scanning software (Wiz, Crowdstrike)

If it's an Azure-owned product (like a "packaged thing") then we have less need for including the extra monitoring and security items because we can "trust Microsoft." If it's self-hosted, we have to treat it like a standard Kubernetes cluster and include all the extra bits.

Probably good if we pick up over there to avoid confusion.
kubefleet-dev/kubefleet#290

[Divya1388]

Hi @ryanzhang-oss
Thanks for looking into this!
As Travis mentioned, things are a bit confusing due to the presence of two repositories.
For the issues outlined above, we've conducted a POC using both:

• https://github.com/Azure/fleet/tree/v0.16.5
• https://github.com/kubefleet-dev/kubefleet

Regarding your question I wonder if you have any reason to run the pod in the hub cluster?
There are couple considerations:

1. As a regulated industry, we need to install enterprise monitoring software on each of the clusters we provision.

2. We've observed the following behavior:

   • When creating a CRP with a namespace that does not start with fleet-* or kube-*, the objects are created in member clusters, but the deployments fail in the hub.

   • Conversely, when the namespace does start with fleet-* or kube-*, we encounter the following error:

   

Could you please advise on the recommended strategy for handling this? We're trying to determine the best approach moving forward.
Thanks again!

[mchaves27]

Hi, I'm on the same team as Divya and Travis.

We tested another approach by disabling the webhook while keeping the hub-agent's replicaCount at 1. As expected, without the ValidatingWebhookConfiguration, there was nothing preventing pods from being initiated. The Kubernetes version upgrade completed successfully. However, I'm still unsure about the potential implications of disabling the webhook.

Interestingly, even though the webhook was disabled, its Kubernetes Service was still created. It's unclear if this service is actually required in this scenario.

[krunalhinguu]

Based on our discussion @weng271190436 , I wanted to highlight an issue we're encountering when attempting to delete or unjoin member clusters. When we try to remove a membercluster resource, the deletion fails and we're seeing errors in the hub-agent logs

I1118 16:26:51.614247       1 membercluster/membercluster_validating_webhook.go:57] "Validating webhook member cluster DELETE" memberCluster="/member-0"
I1118 16:26:51.618015       1 rest/request.go:1632] body was not decodable (unable to check for Status): couldn't get version/kind; json parse error: json: cannot unmarshal string into Go value of type struct { APIVersion string "json:\"apiVersion,omitempty\""; Kind string "json:\"kind,omitempty\"" }
E1118 16:26:51.618536       1 membercluster/membercluster_validating_webhook.go:61] "Failed to list internalServiceExportList when validating" err="no matches for kind \"InternalServiceExport\" in version \"networking.fleet.azure.com/v1alpha1\""
I1118 16:26:52.067456       1 fleetresourcehandler/fleetresourcehandler_webhook.go:88] "handling namespaced resource in fleet reserved namespaces" GVK="apps/v1, Kind=DaemonSet" namespacedName="kube-system/azure-ip-masq-agent" operation="UPDATE" subResource=""

This is a critical concern for our operations going forward. If we need to deprecate or decommission clusters and remove their support from the hub-agent by deleting the MemberCluster resource, this could become a significant blocker.

We've already attempted several troubleshooting steps, including patching the webhook and removing the CRD, but neither approach has resolved the issue. Could you please take a look at this when you have a chance?

cc - @Divya1388 @ryanzhang-oss

[ryanzhang-oss]

@krunalhinguu , thank you very much for your interests in fleet. I wonder if you can clarify on how did you create the fleet? Are you creating an upstream fleet using kubefleet or you are using the Azure fleet manager? The error you got seems to indicate that you use kubefleet. This error is caused by the related network agents and their CRD (https://github.com/Azure/fleet-networking/) are not installed.

[ryanzhang-oss]

I think we need more sessions to go over all the issues.
btw, @tillig, we now have exactly what you were looking for a few months back. We now allow you to have a separte rollout for a deployment inside a namespace. You can now place namespaced resources individualy. @Arvindthiru can give you a session on exactly how that works

[krunalhinguu]

@ryanzhang-oss Thank you for response.

That's correct. We are using the upstream fleet using kubefleet for our implementation. Regarding the networking CRDs, We assumed they were optional and therefore didn't install them.

Could you confirm if there's a way to handle this without installing fleet-networking, or if these CRDs are mandatory for basic fleet operations?

[ryanzhang-oss]

it will be better to install the network agents since the hub agent assumed that we have the network agents. However, we designed the leave behavior with an opt out option.

https://github.com/kubefleet-dev/kubefleet/blob/4ce199e9d99bc961c39fe95cdd0593002cf208a0/apis/cluster/v1beta1/membercluster_types.go#L100C2-L100C16

if you set the deleteOption as
// DeleteValidationModeSkip skips the validation when deleting a MemberCluster.
DeleteValidationModeSkip = "Skip"

then we will not check the networking CRD.

[krunalhinguu]

Thanks for pointing this out, @ryanzhang-oss. This is very helpful.

Just to confirm, is this what you're suggesting?

apiVersion: cluster.kubernetes-fleet.io/v1beta1
kind: MemberCluster
metadata:
  name: <member-cluster-name>
spec:
  identity:
    name: <name>
    kind: ServiceAccount
    namespace: <namespace>
    apiGroup: ""
  heartbeatPeriodSeconds: 15
  deleteOptions:
    validationMode: "Skip"

[Divya1388]

@krunalhinguu

My understanding is that @weng271190436 suggested using the Azure Fleet repository for our installations instead of the upstream Fleet. @ryanzhang-oss and @weng271190436 , could you please confirm which Helm chart we should be installing?

We have demonstrated how we pull Helm charts and push them to our internal Helm repositories over a call. However, the upstream repository presents the following challenges:

1. Lack of tags
2. Extensive use of symlinks
3. Lack of support for beta1 APIs in hub agent CRD's
4. Presence of conflicting APIs, which we have to manually clean up

These issues are making the process more complex for us.