Soil deployment fails on Azure

[Suse-KevinKlinger]

What happened:

I'm trying to deploy into Azure/AWS with a soil on Azure.
Unfortunately during the gardencontent/seeds/soils step, the deployment times out.
If i do a kubectl --kubeconfig=./export/kube-apiserver/kubeconfig -n garden describe seeds azure-soil, I can see the following error message:
failed to apply manifests: 1 error occurred: could not apply object of kind "PriorityClass" "/fluent-bit": PriorityClass.scheduling.k8s.io "fluent-bit" is invalid: Value: Forbidden: may not be changed in an update.

I'm kinda stuck here. I tried to find out where this comes from but was not able to find a solution.

What you expected to happen:

The deployment should successfully execute step gardencontent/seeds/soils .

How to reproduce it (as minimally and precisely as possible):

I have a AKS cluster with K8s version 1.20.9 and 10 nodes Standard_DS2_v2.
On my local machine I've checked out garden-setup with version v3.18.0 and the Kubernetes autoscaler on commit f2a5f9d.
I deployed the VPA and ran sow deploy -A. I've retried this a couple of times to check out if it's a one time error.

Here's my anonymized acre.yaml:

landscape:
  name: kk-gardener
  domain: kksrs.de

  cluster:
    networks:
      nodes: 10.240.0.4/20
      pods: 10.244.0.0/16
      services: 10.0.0.0/16

  iaas:
    - name: azure-soil
      type: azure
      mode: soil
      cloudprofile: azure
      region: westeurope
      zones:
        - westeurope
      credentials:
        clientID: <clientID>
        clientSecret: <clientSecret>
        subscriptionID: <subscriptionID>
        tenantID: <tenantID>
      seeds:
         - name: azure-seed
           type: azure
           mode: seed
           region: westeurope
           credentials:
                clientID: <clientID>
                clientSecret: <clientSecret>
                subscriptionID: <subscriptionID>
                tenantID: <tenantID>
           cluster:
             networks:
               nodes: 10.222.0.0/16
               pods: 10.223.0.0/16
               services: 10.223.0.0/16
               vnet:
                 cidr: 10.222.0.0/16
               workers: 10.222.0.0/19
         - name: aws-seed
           type: aws
           mode: seed
           region: eu-west-1
           zones:
             - eu-west-1a
             - eu-west-1b
             - eu-west-1c
           credentials:
                accessKeyID: <accessKeyID>
                secretAccessKey: <secretAccessKey>
           cluster:
             networks:
               internal: 10.242.112.0/22
               nodes: 10.242.0.0/16
               pods: 10.243.128.0/17
               public: 10.242.96.0/22
               services: 10.243.0.0/17
               vpc:
                 cidr: 10.242.0.0/16
               workers: 10.242.0.0/19

  etcd:
    backup:
      type: abs
      resourceGroup: kklinger-gardener-research
      credentials:
        clientID: <clientID>
        clientSecret: <clientSecret>
        subscriptionID: <subscriptionID>
        tenantID: <tenantID>

  dns:
    type: azure-dns
    credentials:
        clientID: <clientID>
        clientSecret: <clientSecret>
        subscriptionID: <subscriptionID>
        tenantID: <tenantID>

  identity:
    users:
      - email: <secretUser>
        username: <secretUsername>
        password: <secretPW>

  cert-manager:
    email: kevin.klinger@suse.com
    server:
        url: self-signed

Environment:

• Version of garden-setup (release or commit)
  • v3.18.0
• Versions of components (only needed if you overwrote the defaults)
  • VPA on commit f2a5f9d
• Where does the underlying base cluster come from and which operating system does it use?
  • AKS with K8s v1.20.9
• Which cloud provider is configured for the setup?
  • Azure & AWS

[Suse-KevinKlinger]

Hello,

I have further noticed, that the gardenlet Pod was restarted automatically and if I check the logs , I can see the following message over and over again:

{"level":"error","msg":"Seed  bootstrapping failed: failed to apply manifests: 1 error occurred:  could not apply object of kind \"PriorityClass\" \"/fluent-bit\":  PriorityClass.scheduling.k8s.io \"fluent-bit\" is invalid: Value:  Forbidden: may not be changed in an  update.","seed":"azure-soil","time":"2021-11-30T08:46:36Z"}