Mutating webhook preventing pod deletion during deployment rollout.

[Bakies]

I'm not really sure how I got to this state. The kubernetes garbage collector is trying to delete a pod while the mutating webhook is trying to add the FLAGD env vars. This is preventing the deletion of the pod since a pod's env cannot be updated. My theory is that the rs controller was trying to manipulate the finalizers or something for the garbage collection process, but the mutating webhook also wanted to add this when the UPDATE happened.

kube-controller-manager-cluster-r8d4j kube-controller-manager E0929 19:43:00.862321       1 garbagecollector.go:360] "Unhandled Error" err=<
kube-controller-manager-cluster-r8d4j kube-controller-manager 	error syncing item &garbagecollector.node{identity:garbagecollector.objectReference{OwnerReference:v1.OwnerReference{APIVersion:"v1", Kind:"Pod", Name:"deploy-668467c964-pzg2f", UID:"18dcc8e3-0543-4f02-8a09-f077fcaab676", Controller:(*bool)(nil), BlockOwnerDeletion:(*bool)(nil)}, Namespace:"app-name"}, dependentsLock:sync.RWMutex{w:sync.Mutex{state:0, sema:0x0}, writerSem:0x0, readerSem:0x0, readerCount:atomic.Int32{_:atomic.noCopy{}, v:1}, readerWait:atomic.Int32{_:atomic.noCopy{}, v:0}}, dependents:map[*garbagecollector.node]struct {}{}, deletingDependents:true, deletingDependentsLock:sync.RWMutex{w:sync.Mutex{state:0, sema:0x0}, writerSem:0x0, readerSem:0x0, readerCount:atomic.Int32{_:atomic.noCopy{}, v:0}, readerWait:atomic.Int32{_:atomic.noCopy{}, v:0}}, beingDeleted:true, beingDeletedLock:sync.RWMutex{w:sync.Mutex{state:0, sema:0x0}, writerSem:0x0, readerSem:0x0, readerCount:atomic.Int32{_:atomic.noCopy{}, v:0}, readerWait:atomic.Int32{_:atomic.noCopy{}, v:0}}, virtual:false, virtualLock:sync.RWMutex{w:sync.Mutex{state:0, sema:0x0}, writerSem:0x0, readerSem:0x0, readerCount:atomic.Int32{_:atomic.noCopy{}, v:0}, readerWait:atomic.Int32{_:atomic.noCopy{}, v:0}}, owners:[]v1.OwnerReference{v1.OwnerReference{APIVersion:"apps/v1", Kind:"ReplicaSet", Name:"deploy-668467c964", UID:"6df8db92-3c35-4821-836f-4de19f691af8", Controller:(*bool)(0xc006b59467), BlockOwnerDeletion:(*bool)(0xc006b59468)}}}: Pod "deploy-668467c964-pzg2f" is invalid: spec: Forbidden: pod updates may not change fields other than `spec.containers[*].image`,`spec.initContainers[*].image`,`spec.activeDeadlineSeconds`,`spec.tolerations` (only additions to existing tolerations),`spec.terminationGracePeriodSeconds` (allow it to be set to 1 if it was previously negative)
kube-controller-manager-cluster-r8d4j kube-controller-manager 	  core.PodSpec{
kube-controller-manager-cluster-r8d4j kube-controller-manager 	  	Volumes:        nil,
kube-controller-manager-cluster-r8d4j kube-controller-manager 	  	Containers: []core.Container{
kube-controller-manager-cluster-r8d4j kube-controller-manager 	  		{
kube-controller-manager-cluster-r8d4j kube-controller-manager 	  			... // 5 identical fields
kube-controller-manager-cluster-r8d4j kube-controller-manager 	  			Ports:   {{Name: "server", ContainerPort: 8080, Protocol: "TCP"}},
kube-controller-manager-cluster-r8d4j kube-controller-manager 	  			EnvFrom: {{ConfigMapRef: &{LocalObjectReference: {Name: "deploy-env"}}}},
kube-controller-manager-cluster-r8d4j kube-controller-manager 	  			Env: []core.EnvVar{
kube-controller-manager-cluster-r8d4j kube-controller-manager 	  				... // 13 identical elements
kube-controller-manager-cluster-r8d4j kube-controller-manager 	+ 				{Name: "FLAGD_HOST", Value: "flagd.app-name"},
kube-controller-manager-cluster-r8d4j kube-controller-manager 	+ 				{Name: "FLAGD_PORT", Value: "8015"},
kube-controller-manager-cluster-r8d4j kube-controller-manager 	+ 				{Name: "FLAGD_TLS", Value: "false"},
kube-controller-manager-cluster-r8d4j kube-controller-manager 	+ 				{Name: "FLAGD_CACHE", Value: "lru"},
kube-controller-manager-cluster-r8d4j kube-controller-manager 	+ 				{Name: "FLAGD_MAX_CACHE_SIZE", Value: "1000"},
kube-controller-manager-cluster-r8d4j kube-controller-manager 	+ 				{Name: "FLAGD_RESOLVER", Value: "in-process"},
kube-controller-manager-cluster-r8d4j kube-controller-manager 	  				{Name: "FLAGD_HOST", Value: "flagd.app-name"},
kube-controller-manager-cluster-r8d4j kube-controller-manager 	  				{Name: "FLAGD_PORT", Value: "8015"},
kube-controller-manager-cluster-r8d4j kube-controller-manager 	  				... // 4 identical elements
kube-controller-manager-cluster-r8d4j kube-controller-manager 	  			},
kube-controller-manager-cluster-r8d4j kube-controller-manager 	  			Resources:    {Limits: {s"memory": {i: {...}, s: "2Gi", Format: "BinarySI"}}, Requests: {s"cpu": {i: {...}, s: "100m", Format: "DecimalSI"}, s"memory": {i: {...}, Format: "BinarySI"}}},
kube-controller-manager-cluster-r8d4j kube-controller-manager 	  			ResizePolicy: nil,
kube-controller-manager-cluster-r8d4j kube-controller-manager 	  			... // 14 identical fields
kube-controller-manager-cluster-r8d4j kube-controller-manager 	  		},
kube-controller-manager-cluster-r8d4j kube-controller-manager 	  	},
kube-controller-manager-cluster-r8d4j kube-controller-manager 	  	EphemeralContainers: nil,
kube-controller-manager-cluster-r8d4j kube-controller-manager 	  	RestartPolicy:       "Always",
kube-controller-manager-cluster-r8d4j kube-controller-manager 	  	... // 29 identical fields
kube-controller-manager-cluster-r8d4j kube-controller-manager 	  }
kube-controller-manager-cluster-r8d4j kube-controller-manager  > logger="UnhandledError"

[thisthat]

Hi @Bakies, is this easy to reproduce on your end?

@toddbaert, @beeme1mr looking at the code of OFO, we never look into the operation type 🤔
We could add if req.Operation == admissionv1.Delete { return admission.Allowed("deletion - no patching requried") } even before parsing the request 🤔