kubeconfigs for ManagedSeeds not working anymore after garden-setup update

[christianhuening]

What happened:

I played around with garden-setup and tried updating a garden setup from 3.14.0 to 3.15.0 which had non-production seeds and shoots. After re-using sow deploy -A everything was fine, but we discovered that the generated kubeconfigs for the seed clusters would give us “unauthorized” errors. So we tried to rotate them and found that then they issue a “certificate mismatch” error. (the kubeconfigs for the Shoots kept working)

I discovered is that the ca secret in the shoot control plane namespace staid in fact the same. The only thing that mysteriously changed is the CA backed into the kubecfg secret in that namespace. It’s just a different one. If I manually replace it with the cacontent, it of course works again. The api server kept using the ca one naturally.

What you expected to happen:

CA entries in kubeconfigs for seed clusters are not changed.

How to reproduce it (as minimally and precisely as possible):

1. Deploy a garden-setup 3.14 garden onto GKE that also is a Seed cluster.
2. Deploy a seed into it (for/on AWS in case it matters).
3. Update garden-setup to 3.15 and re-run sow deploy -A

Environment:

• Version of garden-setup (release or commit): 3.14 -> 3.15
• Versions of components (only needed if you overwrote the defaults)
• Where does the underlying base cluster come from and which operating system does it use? GKE & COS
• Which cloud provider is configured for the setup? GCP

[christianhuening]

I just re-encountered that when upgrading from 1.29.x to 1.36.x. This time though the issue is with the gardener-resource-manager. The gardener-resource-manager-server secret uses a CA which is 6 seconds older than the one included in the ca secret in the shoot's namespace. Hence the webhook stops working with the apiserver preventing shoot control planes from functioning.