Syncer fails to create pods on host when topologySpreadConstraints uses matchLabelKeys (k8s 1.34+ API server mutation conflict)

[llavaud]

What happened?

Pods with topologySpreadConstraints.matchLabelKeys defined in their Deployment spec fail to sync to the host cluster with the following error:

spec.topologySpreadConstraints[0][0]: Invalid value: "pod-template-hash":
exists in both matchLabelKeys and labelSelector

Starting with Kubernetes v1.34, the API server mutates pods at creation time via mutateTopologySpreadConstraints() in PrepareForCreate (pkg/registry/core/pod/strategy.go). When the feature gates MatchLabelKeysInPodTopologySpread and MatchLabelKeysInPodTopologySpreadSelectorMerge are both enabled (both Beta/on-by-default since v1.34), the API server resolves matchLabelKeys values into labelSelector.matchExpressions using the pod's own labels.

This causes a conflict with the syncer's pod creation flow:

1. A Deployment inside the vcluster defines matchLabelKeys: ["pod-template-hash"] with a labelSelector that does not contain pod-template-hash.
2. The virtual API server (k8s 1.35) mutates the pod at creation → injects pod-template-hash into labelSelector.matchExpressions while keeping matchLabelKeys in the spec. This is expected k8s 1.34+ behavior.
3. The syncer reads the mutated pod spec and creates it as-is on the host cluster.
4. The host API server receives a CREATE request where pod-template-hash already exists in both matchLabelKeys and labelSelector. The validation added in kubernetes/kubernetes#130614 rejects this as invalid.

On a real k8s 1.35 cluster (without vcluster), this never happens because the pod is only created once — the API server mutates and validates in a single flow. The double-CREATE pattern (virtual API server → syncer → host API server) causes the host to see a pre-mutated spec that fails validation.

The relevant syncer code is in pkg/controllers/resources/pods/translate/translator.go (translateTopologySpreadConstraints), which copies the labelSelector as-is without stripping entries that were added by the virtual API server's mutateTopologySpreadConstraints.

Upstream references:

• kubernetes/kubernetes#129480 — Design issue: align TSC matchLabelKeys with PodAffinity
• kubernetes/kubernetes#129874 — Implementation of API server mutation (merged for v1.34)
• kubernetes/enhancements#5205 — KEP-3243 update with feature gate MatchLabelKeysInPodTopologySpreadSelectorMerge

Workaround: Disable the feature gate on the virtual API server:

controlPlane:
  distro:
    k8s:
      apiServer:
        extraArgs:
          - "--feature-gates=MatchLabelKeysInPodTopologySpreadSelectorMerge=false"

What did you expect to happen?

Pods with matchLabelKeys in topologySpreadConstraints should be synced to the host cluster successfully. The syncer should account for the k8s 1.34+ API server mutation by stripping the resolved matchLabelKeys entries from labelSelector.matchExpressions before creating the pod on the host (letting the host's own mutation re-add them), or by clearing matchLabelKeys from the synced pod spec entirely.

How can we reproduce it (as minimally and precisely as possible)?

1. Create a vcluster v0.32.1 with k8s 1.34+ on a host cluster running k8s 1.34+:

vcluster create test --kubernetes-version v1.35.0

2. Inside the vcluster, create a Deployment with matchLabelKeys in topologySpreadConstraints:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: test-tsc
spec:
  replicas: 1
  selector:
    matchLabels:
      app: test-tsc
  template:
    metadata:
      labels:
        app: test-tsc
    spec:
      containers:
        - name: nginx
          image: nginx:alpine
      topologySpreadConstraints:
        - maxSkew: 1
          topologyKey: kubernetes.io/hostname
          whenUnsatisfiable: ScheduleAnyway
          labelSelector:
            matchLabels:
              app: test-tsc
          matchLabelKeys:
            - pod-template-hash

3. Observe that the pod stays Pending inside the vcluster. Check events:

kubectl get events --field-selector reason=SyncError

Expected output:

Warning  SyncError  pod/test-tsc-xxx  Error syncing to host cluster: create object:
Pod "..." is invalid: spec.topologySpreadConstraints[0][0]: Invalid value:
"pod-template-hash": exists in both matchLabelKeys and labelSelector

4. Verify the pod spec inside the vcluster shows the mutation (matchLabelKeys resolved into labelSelector):

kubectl get pod <pod-name> -o jsonpath='{.spec.topologySpreadConstraints[0].labelSelector}'

This will show pod-template-hash in matchExpressions, which was not in the Deployment template — it was added by the virtual API server's mutateTopologySpreadConstraints.

Anything else we need to know?

No response

Host cluster Kubernetes version

Details

Client Version: v1.35.0
Kustomize Version: v5.7.1
Server Version: v1.35.0-33+37970203ae1a44

vcluster version

Details

vcluster version 0.32.1

VCluster Config

Details

        sync:
          fromHost:
            priorityClasses:
              enabled: true
          toHost:
            serviceAccounts:
              enabled: true
        controlPlane:
          distro:
            k8s:
              image:
                tag: v1.35.0
              apiServer:
                extraArgs:
                  - --feature-gates=VolumeAttributesClass=true
                  - --runtime-config=storage.k8s.io/v1beta1=true
                  - --request-timeout=120s
                  - --feature-gates=MatchLabelKeysInPodTopologySpreadSelectorMerge=false
              controllerManager:
                extraArgs:
                  - --feature-gates=VolumeAttributesClass=true
          statefulSet:
            labels:
              finops.mirakl.net/product: tools
              finops.mirakl.net/service: vcluster
            pods:
              labels:
                finops.mirakl.net/product: tools
                finops.mirakl.net/service: vcluster
            scheduling:
              priorityClassName: user-cluster-critical
            resources:
              limits:
                memory: 6Gi
              requests:
                memory: 1Gi
                cpu: 500m