-
Notifications
You must be signed in to change notification settings - Fork 974
Description
What happened:
When an environment variable (e.g., NODE_ENV
) appears multiple times in the env
array in a Kubernetes manifest, and one duplicate is removed, running kubectl apply
will remove all instances of that variable from the live resource. Only after a subsequent apply (with the deduplicated manifest) will the variable be added again.
This is a breaking change compared to prior behavior, where the first instance was preserved. In modern deployment pipelines, manifests are often generated automatically, and such duplicities can easily occur.
Important
This change fundamentally alters the behavior of kubectl apply
regarding duplicate environment variables—a core functionality used daily by thousands of users for over a decade.
Why this is a critical problem:
- Automated toolchains (Helm, Helmfile, ArgoCD, Flux, GitHub Actions, CI/CD, GitOps, etc.) often generate or process manifests where accidental duplicates are possible and hard to eliminate completely.
- Workaround ("use server-side apply") is not acceptable – many toolchains, including Helm and ArgoCD, rely on the default client-side apply and cannot easily switch to server-side apply.
- This behavior can cause outages or severe misconfiguration in production, as critical environment variables may be silently removed from running workloads.
- Detection is difficult: The issue is not obvious and may lead to silent failures, security issues, or outages if environment variables disappear unexpectedly.
- This is not backwards compatible with previous
kubectl
/Kubernetes behavior and is a regression for many users.
How to reproduce it (minimized):
apiVersion: apps/v1
kind: Deployment
metadata:
name: nginx
spec:
selector:
matchLabels:
app: nginx
template:
metadata:
labels:
app: nginx
spec:
containers:
- name: nginx
image: nginx
env:
- name: NODE_ENV
value: "qa"
- name: NODE_ENV
value: "qa"
kubectl apply -f deployment.yaml
# creates deployment with duplicated env- Remove one of the
NODE_ENV
envs from the manifest. kubectl apply -f deployment.yaml
# both NODE_ENV are removed from the live objectkubectl apply -f deployment.yaml
# only now is NODE_ENV re-added
Expected behavior:
- Removing one duplicate should only remove that instance; at least one instance of the env var should remain.
- Ideally, apply should error on duplicate keys in lists, or at minimum, preserve the first occurrence, and never silently remove all.
Environment:
- Kubernetes Client Version:
v1.30.5
- Kubernetes Server Version:
v1.30-v1.33
- OS: Linux
Why the current workaround is insufficient
- Not all toolchains support server-side apply.
- Helm, Helmfile, ArgoCD and many others cannot simply be reconfigured.
- Manual linting or deduplication is not always feasible in complex CI/CD environments.
Request
- Please consider this a critical regression and re-evaluate the merge/diff algorithm.
- At minimum,
kubectl apply
should fail with a clear error on duplicated items in lists (such as env). - Never silently remove all instances of a variable if just one was removed from the input manifest.
Related
- Prior closed issue: kubectl apply incorrectly deletes duplicate environment variables #1750 (closed with "workaround" that is not suitable for most real-world use cases)
This bug can cause production outages and is not a corner case. Please re-open or escalate.