kubectl diff --server-side does not detect deletions

[gnunn1]

What happened:

The command kubectl diff --server-side fails to detect fields that are removed from the manifest even though kubectl apply reconciles the removal correctly. Using kubectl diff without the --server-side switch works fine since it leverages the last-applied-configuration annotation.

What you expected to happen:

The kubectl diff --server-side command should align with kubectl apply --server-side which AFAIK uses the structured-merge-diff library to do its work. Otherwise the diff is generating output that is IMHO errorneous and goes against user expectations that diff should output what the effects of an apply will be.

Ideally diff --server-side should use the structured-merge-diff library rather then relying on an alternate way to generate the diff.

How to reproduce it (as minimally and precisely as possible):

Save this deployment as a file:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: basic-app
  labels:
    test: value
  annotations:
    test: value
    test2: value
spec:
  replicas: 1
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      app: basic-app
  template:
    metadata:
      labels:
        app: basic-app
    spec:
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
              - matchExpressions:
                  - key: node-role.kubernetes.io/worker
                    operator: In
                    values:
                      - ''
                  - key: kubernetes.io/os
                    operator: In
                    values:
                      - 'linux'
      containers:
      - name: basic-app
        image: busybox:latest # A simple image for demonstration
        command: ["sh", "-c", "echo $APP_MESSAGE && echo 'Application running on port:' $APP_PORT && sleep 3600"]
        env:
        - name: TEST_VAR
          value: value
        - name: TEST_VAR2
          value: value2

And then apply it to the cluster:

kubectl apply -f deployment.yaml

Remove the affinity section from the manifest (but note it is the same issue regardless of what you remove) and then diff it with server side:

$ kubectl diff -f deployment.yaml --server-side
diff -u -N /tmp/LIVE-486847088/apps.v1.Deployment.diff-bug.basic-app /tmp/MERGED-2682306472/apps.v1.Deployment.diff-bug.basic-app
--- /tmp/LIVE-486847088/apps.v1.Deployment.diff-bug.basic-app   2025-11-05 13:18:43.473757610 -0800
+++ /tmp/MERGED-2682306472/apps.v1.Deployment.diff-bug.basic-app        2025-11-05 13:18:43.474757618 -0800
@@ -4,11 +4,11 @@
   annotations:
     deployment.kubernetes.io/revision: "1"
     kubectl.kubernetes.io/last-applied-configuration: |
-      {"apiVersion":"apps/v1","kind":"Deployment","metadata":{"annotations":{"test":"value","test2":"value"},"labels":{"test":"value"},"name":"basic-app","namespace":"diff-bug"},"spec":{"replicas":1,"selector":{"matchLabels":{"app":"basic-app"}},"template":{"metadata":{"labels":{"app":"basic-app"}},"spec":{"affinity":{"nodeAffinity":{"requiredDuringSchedulingIgnoredDuringExecution":{"nodeSelectorTerms":[{"matchExpressions":[{"key":"node-role.kubernetes.io/worker","operator":"In","values":[""]},{"key":"kubernetes.io/os","operator":"In","values":["linux"]}]}]}}},"containers":[{"command":["sh","-c","echo $APP_MESSAGE \u0026\u0026 echo 'Application running on port:' $APP_PORT \u0026\u0026 sleep 3600"],"env":[{"name":"TEST_VAR","value":"value"},{"name":"TEST_VAR2","value":"value2"}],"image":"busybox:latest","name":"basic-app"}]}}}}
+      {"apiVersion":"apps/v1","kind":"Deployment","metadata":{"annotations":{"test":"value","test2":"value"},"labels":{"test":"value"},"name":"basic-app","namespace":"diff-bug"},"spec":{"replicas":1,"selector":{"matchLabels":{"app":"basic-app"}},"template":{"metadata":{"labels":{"app":"basic-app"}},"spec":{"containers":[{"command":["sh","-c","echo $APP_MESSAGE \u0026\u0026 echo 'Application running on port:' $APP_PORT \u0026\u0026 sleep 3600"],"env":[{"name":"TEST_VAR","value":"value"},{"name":"TEST_VAR2","value":"value2"}],"image":"busybox:latest","name":"basic-app"}]}}}}
     test: value
     test2: value
   creationTimestamp: "2025-11-05T21:18:15Z"
-  generation: 1
+  generation: 2
   labels:
     test: value
   name: basic-app

The diff fails to show the removed affinity. Diff'ing this using client-side diff works fine since it leverages the last-applied-configuration. Also running kubectl apply -f deployment.yaml updates the manifest as expected and removes the affinity section using either client-side or server-side apply.

Anything else we need to know?:

Tools like Argo CD rely on the way Kubernetes handles the diff and this dissonance between diff and apply causes these tools to fail to report differences as expected to their users.

Environment:

• Kubernetes client and server versions (use kubectl version):

Client Version: v1.34.1
Kustomize Version: v5.7.1
Server Version: v1.33.5

• Cloud provider or hardware configuration:

on-prem

• OS (e.g: cat /etc/os-release):

NAME="Arch Linux"
PRETTY_NAME="Arch Linux"
ID=arch
BUILD_ID=rolling
ANSI_COLOR="38;2;23;147;209"
HOME_URL="https://archlinux.org/"
DOCUMENTATION_URL="https://wiki.archlinux.org/"
SUPPORT_URL="https://bbs.archlinux.org/"
BUG_REPORT_URL="https://gitlab.archlinux.org/groups/archlinux/-/issues"
PRIVACY_POLICY_URL="https://terms.archlinux.org/docs/privacy-policy/"
LOGO=archlinux-logo

[k8s-ci-robot]

This issue is currently awaiting triage.

SIG CLI takes a lead on issue triage for this repo, but any Kubernetes member can accept issues by applying the triage/accepted label.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[gnunn1]

FYI @leoluz

[leoluz]

We had a similar issue when we implemented support for SSA in Argo CD. The diff logic was previously only relying on the last-applied-configuration annotation which caused inconsistent diffs when SSA was enabled. We tried to mimic what k8s does on the server-side leveraging the structured-merge-diff library to support the logic to generate the diffs. However, that attempt was unsuccessful. What we noticed is that SMD depends on the CRD schemas to be well defined. Extracting the schemas using available k8s APIs returned them in a "non complete state". Things like default values defined at the CRD level weren't being provided which directly impacted the diff results.

I am curious if it is technically possible to apply the same merge logic that happens in k8s when SSA is enable but entirely on the client side (Kubectl or Argo CD for example). Admission controllers won't be executed for sure but maybe that is fine for the Kubectl use-case.

[ardaguclu]

diff --server-side works properly with apply --server-side. I'd expect some incompatibilities between diff --server-side against the resources that are previously created using client side apply (i.e. kubectl apply -f deployment.yaml).

So did you try applying resource first via kubectl apply -f deployment.yaml --server-side -> remove the fields from yaml -> kubectl diff -f deployment.yaml --server-side?

[ardaguclu]

Also note that server-side apply does not rely on last-applied-configuration at all. This annotation is for client side apply.

[gnunn1]

You are correct, using kubectl apply -f deploy.yaml --server-side, making the change and then running kubectl diff --server-side works in terms of showing the differences. I could have sworn I had tested this before posting but I must have made a mistake. Thanks.