[Bug] The sidecar doesn't work well with env injected by mutation hooks

[mnencia]

After #354, released in version 0.4.1, the sidecar injection conflicts with any MutatingWebhookConfiguration that adds environment variables to all containers. This issue is affecting all EKS users.

This is the error returned by the API server:

cannot updated metadata on pods: Pod "postgres-cluster-example-1" is invalid: spec: Forbidden: pod updates may not change fields other than `spec.containers[*].image`,`spec.initContainers[*].image`,`spec.activeDeadlineSeconds`,`spec.tolerations` (only additions to existing tolerations),`spec.terminationGracePeriodSeconds` (allow it to be set to 1 if it was previously negative)
  core.PodSpec{
  	Volumes: {{Name: "eks-pod-identity-token", VolumeSource: {Projected: &{Sources: {{ServiceAccountToken: &{Audience: "pods.eks.amazonaws.com", ExpirationSeconds: 80277, Path: "eks-pod-identity-token"}}}, DefaultMode: &420}}}, {Name: "pgdata", VolumeSource: {PersistentVolumeClaim: &{ClaimName: "postgres-cluster-example-1"}}}, {Name: "scratch-data", VolumeSource: {EmptyDir: &{}}}, {Name: "shm", VolumeSource: {EmptyDir: &{Medium: "Memory"}}}, ...},
  	InitContainers: []core.Container{
  		{Name: "bootstrap-controller", Image: "ghcr.io/cloudnative-pg/cloudnative-pg:1.26.0", Command: {"/manager", "bootstrap", "/controller/manager", "--log-level=info"}, Env: {{Name: "AWS_STS_REGIONAL_ENDPOINTS", Value: "regional"}, {Name: "AWS_DEFAULT_REGION", Value: "us-east-1"}, {Name: "AWS_REGION", Value: "us-east-1"}, {Name: "AWS_CONTAINER_CREDENTIALS_FULL_URI", Value: "http://169.254.170.23/v1/credentials"}, ...}, ...},
  		{
  			... // 5 identical fields
  			Ports:   nil,
  			EnvFrom: nil,
  			Env: []core.EnvVar{
  				... // 6 identical elements
  				{Name: "PGHOST", Value: "/controller/run"},
  				{Name: "TMPDIR", Value: "/controller/tmp"},
- 				{Name: "SPOOL_DIRECTORY", Value: "/controller/wal-restore-spool"},
- 				{Name: "CUSTOM_CNPG_GROUP", Value: "postgresql.cnpg.io"},
- 				{Name: "CUSTOM_CNPG_VERSION", Value: "v1"},
  				{Name: "AWS_STS_REGIONAL_ENDPOINTS", Value: "regional"},
  				{Name: "AWS_DEFAULT_REGION", Value: "us-east-1"},
  				{Name: "AWS_REGION", Value: "us-east-1"},
  				{Name: "AWS_CONTAINER_CREDENTIALS_FULL_URI", Value: "http://169.254.170.23/v1/credentials"},
  				{Name: "AWS_CONTAINER_AUTHORIZATION_TOKEN_FILE", Value: "/var/run/secrets/pods.eks.amazonaws.com/serviceaccount/eks-pod-i"...},
+ 				{Name: "SPOOL_DIRECTORY", Value: "/controller/wal-restore-spool"},
+ 				{Name: "CUSTOM_CNPG_GROUP", Value: "postgresql.cnpg.io"},
+ 				{Name: "CUSTOM_CNPG_VERSION", Value: "v1"},
  			},
  			Resources:    {},
  			ResizePolicy: nil,
  			... // 14 identical fields
  		},
  	},
  	Containers:          {{Name: "postgres", Image: "ghcr.io/cloudnative-pg/postgresql:17.5-standard-bookworm", Command: {"/controller/manager", "instance", "run", "--status-port-tls", ...}, Ports: {{Name: "postgresql", ContainerPort: 5432, Protocol: "TCP"}, {Name: "metrics", ContainerPort: 9187, Protocol: "TCP"}, {Name: "status", ContainerPort: 8000, Protocol: "TCP"}}, ...}},
  	EphemeralContainers: nil,
  	... // 30 identical fields
  }

When creating the sidecar configuration, we should refrain from altering the order of environment variables if all required variables are already included in the configuration.

[Utwo]

I've updated to 0.5.0 and I still got this error from cloudnative-pg controller pod. I'm on EKS using eks-pod-identity addon.

Cannot updated metadata on pods: Pod \"airflow-db-3\" is invalid: spec: Forbidden: pod updates may not change fields other than

[tholinka]

On 0.5.0 (Talos cluster with k8tz operator injecting env variables):

operator logs:

{"level":"info","ts":"2025-06-03T17:55:17.416609635-05:00","msg":"Pod rollout required","controller":"cluster","controllerGroup":"postgresql.cnpg.io","controllerKind":"Cluster","Cluster":{"name":"postgres-cluster","namespace":"database"},"namespace":"database","name":"postgres-cluster","reconcileID":"9759fb91-99b9-484d-9868-49ac8554f329","podName":"postgres-cluster-1","reason":"original and target PodSpec differ in volumes: element barman-certificates has been removed"}
{"level":"info","ts":"2025-06-03T17:55:17.447887295-05:00","msg":"Cluster has become unhealthy","controller":"cluster","controllerGroup":"postgresql.cnpg.io","controllerKind":"Cluster","Cluster":{"name":"postgres-cluster","namespace":"database"},"namespace":"database","name":"postgres-cluster","reconcileID":"9759fb91-99b9-484d-9868-49ac8554f329"}
{"level":"info","ts":"2025-06-03T17:55:17.447934286-05:00","msg":"Recreating instance pod","controller":"cluster","controllerGroup":"postgresql.cnpg.io","controllerKind":"Cluster","Cluster":{"name":"postgres-cluster","namespace":"database"},"namespace":"database","name":"postgres-cluster","reconcileID":"9759fb91-99b9-484d-9868-49ac8554f329","pod":"postgres-cluster-1","to":"ghcr.io/cloudnative-pg/postgresql:17.5-standard-bookworm@sha256:8bcc3eb5b0b2dd7ad91c3c57eadc0df37148f28be16fc8062ed5eb29a2a4a4b7","reason":"Restarting instance postgres-cluster-1, because: original and target PodSpec differ in volumes: element barman-certificates has been removed"}
{"level":"info","ts":"2025-06-03T17:55:30.029875919-05:00","msg":"Creating new Pod to reattach a PVC","controller":"cluster","controllerGroup":"postgresql.cnpg.io","controllerKind":"Cluster","Cluster":{"name":"postgres-cluster","namespace":"database"},"namespace":"database","name":"postgres-cluster","reconcileID":"c44d683d-8e84-45a8-ad22-5649814ada4f","pod":"postgres-cluster-1","pvc":"postgres-cluster-1"}
{"level":"info","ts":"2025-06-03T17:58:50.811439327-05:00","msg":"Cannot extract Pod status","controller":"cluster","controllerGroup":"postgresql.cnpg.io","controllerKind":"Cluster","Cluster":{"name":"postgres-cluster","namespace":"database"},"namespace":"database","name":"postgres-cluster","reconcileID":"2135afb9-ae41-4429-8cb3-010de12dcbc8","podName":"postgres-cluster-1","error":"error status code: 500, body: failed to connect to `user=postgres database=postgres`: /controller/run/.s.PGSQL.5432 (/controller/run): dial error: dial unix /controller/run/.s.PGSQL.5432: connect: no such file or directory\n"}
{"level":"info","ts":"2025-06-03T17:58:50.84080533-05:00","msg":"Setting replica label","controller":"cluster","controllerGroup":"postgresql.cnpg.io","controllerKind":"Cluster","Cluster":{"name":"postgres-cluster","namespace":"database"},"namespace":"database","name":"postgres-cluster","reconcileID":"2135afb9-ae41-4429-8cb3-010de12dcbc8","pod":"postgres-cluster-1"}
{"level":"error","ts":"2025-06-03T17:58:50.879441053-05:00","msg":"while patching instance metadata","controller":"cluster","controllerGroup":"postgresql.cnpg.io","controllerKind":"Cluster","Cluster":{"name":"postgres-cluster","namespace":"database"},"namespace":"database","name":"postgres-cluster","reconcileID":"2135afb9-ae41-4429-8cb3-010de12dcbc8","instanceName":"postgres-cluster-1","error":"Pod \"postgres-cluster-1\" is invalid: spec: Forbidden: pod updates may not change fields other than `spec.containers[*].image`,`spec.initContainers[*].image`,`spec.activeDeadlineSeconds`,`spec.tolerations` (only additions to existing tolerations),`spec.terminationGracePeriodSeconds` (allow it to be set to 1 if it was previously negative)\n  core.PodSpec{\n  \tVolumes: {{Name: \"pgdata\", VolumeSource: {PersistentVolumeClaim: &{ClaimName: \"postgres-cluster-1\"}}}, {Name: \"scratch-data\", VolumeSource: {EmptyDir: &{}}}, {Name: \"shm\", VolumeSource: {EmptyDir: &{Medium: \"Memory\"}}}, {Name: \"plugins\", VolumeSource: {EmptyDir: &{}}}, ...},\n  \tInitContainers: []core.Container{\n  \t\t{Name: \"bootstrap-controller\", Image: \"ghcr.io/cloudnative-pg/cloudnative-pg:1.26.0\", Command: {\"/manager\", \"bootstrap\", \"/controller/manager\", \"--log-level=info\"}, Resources: {Limits: {s\"hugepages-2Mi\": {i: {...}, s: \"1Gi\", Format: \"BinarySI\"}, s\"memory\": {i: {...}, s: \"2Gi\", Format: \"BinarySI\"}}, Requests: {s\"cpu\": {i: {...}, s: \"500m\", Format: \"DecimalSI\"}, s\"hugepages-2Mi\": {i: {...}, s: \"1Gi\", Format: \"BinarySI\"}, s\"memory\": {i: {...}, s: \"2Gi\", Format: \"BinarySI\"}}}, ...},\n  \t\t{\n  \t\t\t... // 5 identical fields\n  \t\t\tPorts:   nil,\n  \t\t\tEnvFrom: nil,\n  \t\t\tEnv: []core.EnvVar{\n  \t\t\t\t... // 6 identical elements\n  \t\t\t\t{Name: \"PGHOST\", Value: \"/controller/run\"},\n  \t\t\t\t{Name: \"TMPDIR\", Value: \"/controller/tmp\"},\n+ \t\t\t\t{Name: \"TZ\", Value: \"America/Chicago\"},\n  \t\t\t\t{Name: \"SPOOL_DIRECTORY\", Value: \"/controller/wal-restore-spool\"},\n  \t\t\t\t{Name: \"CUSTOM_CNPG_GROUP\", Value: \"postgresql.cnpg.io\"},\n  \t\t\t\t{Name: \"CUSTOM_CNPG_VERSION\", Value: \"v1\"},\n  \t\t\t},\n  \t\t\tResources:     {},\n  \t\t\tResizePolicy:  nil,\n  \t\t\tRestartPolicy: &\"Always\",\n  \t\t\tVolumeMounts: []core.VolumeMount{\n  \t\t\t\t... // 3 identical elements\n  \t\t\t\t{Name: \"plugins\", MountPath: \"/plugins\"},\n  \t\t\t\t{Name: \"kube-api-access-fkkct\", ReadOnly: true, MountPath: \"/var/run/secrets/kubernetes.io/serviceaccount\"},\n+ \t\t\t\t{Name: \"k8tz\", ReadOnly: true, MountPath: \"/usr/share/zoneinfo\"},\n  \t\t\t},\n  \t\t\tVolumeDevices: nil,\n  \t\t\tLivenessProbe: nil,\n  \t\t\t... // 10 identical fields\n  \t\t},\n  \t\t{Name: \"k8tz\", Image: \"quay.io/k8tz/k8tz:0.18.0\", Args: {\"bootstrap\"}, VolumeMounts: {{Name: \"k8tz\", MountPath: \"/mnt/zoneinfo\"}, {Name: \"kube-api-access-fkkct\", ReadOnly: true, MountPath: \"/var/run/secrets/kubernetes.io/serviceaccount\"}}, ...},\n  \t},\n  \tContainers:          {{Name: \"postgres\", Image: \"ghcr.io/cloudnative-pg/postgresql:17.5-standard-bookworm@sha256:\"..., Command: {\"/controller/manager\", \"instance\", \"run\", \"--status-port-tls\", ...}, Ports: {{Name: \"postgresql\", ContainerPort: 5432, Protocol: \"TCP\"}, {Name: \"metrics\", ContainerPort: 9187, Protocol: \"TCP\"}, {Name: \"status\", ContainerPort: 8000, Protocol: \"TCP\"}}, ...}},\n  \tEphemeralContainers: nil,\n  \t... // 30 identical fields\n  }\n","stacktrace":"github.com/cloudnative-pg/machinery/pkg/log.(*logger).Error\n\tpkg/mod/github.com/cloudnative-pg/machinery@v0.2.0/pkg/log/log.go:125\ngithub.com/cloudnative-pg/cloudnative-pg/pkg/reconciler/instance.ReconcileMetadata\n\tpkg/reconciler/instance/metadata.go:66\ngithub.com/cloudnative-pg/cloudnative-pg/internal/controller.(*ClusterReconciler).reconcile\n\tinternal/controller/cluster_controller.go:399\ngithub.com/cloudnative-pg/cloudnative-pg/internal/controller.(*ClusterReconciler).Reconcile\n\tinternal/controller/cluster_controller.go:220\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\tpkg/mod/sigs.k8s.io/controller-runtime@v0.20.4/pkg/internal/controller/controller.go:119\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\tpkg/mod/sigs.k8s.io/controller-runtime@v0.20.4/pkg/internal/controller/controller.go:334\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\tpkg/mod/sigs.k8s.io/controller-runtime@v0.20.4/pkg/internal/controller/controller.go:294\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2\n\tpkg/mod/sigs.k8s.io/controller-runtime@v0.20.4/pkg/internal/controller/controller.go:255"}

plugin-barman-cloud logs:

{"level":"info","ts":"2025-06-03T17:55:17.409878177-05:00","logger":"lifecycle","msg":"Lifecycle hook reconciliation start"}
{"level":"info","ts":"2025-06-03T17:55:17.410259599-05:00","logger":"lifecycle","msg":"Reconciling pod"}
{"level":"debug","ts":"2025-06-03T17:55:17.411882431-05:00","logger":"plugin-barman-cloud-lifecycle","msg":"generated patch","podName":"postgres-cluster-1","caller":"/workspace/internal/cnpgi/operator/lifecycle.go:266","content":"[{\"op\":\"add\",\"path\":\"/spec/volumes/3\",\"value\":{\"name\":\"plugins\",\"emptyDir\":{}}},{\"op\":\"add\",\"path\":\"/spec/initContainers/1\",\"value\":{\"name\":\"plugin-barman-cloud\",\"image\":\"ghcr.io/cloudnative-pg/plugin-barman-cloud-sidecar:v0.5.0\",\"args\":[\"instance\"],\"env\":[{\"name\":\"PGDATA\",\"value\":\"/var/lib/postgresql/data/pgdata\"},{\"name\":\"POD_NAME\",\"value\":\"postgres-cluster-1\"},{\"name\":\"NAMESPACE\",\"value\":\"database\"},{\"name\":\"CLUSTER_NAME\",\"value\":\"postgres-cluster\"},{\"name\":\"PSQL_HISTORY\",\"value\":\"/controller/tmp/.psql_history\"},{\"name\":\"PGPORT\",\"value\":\"5432\"},{\"name\":\"PGHOST\",\"value\":\"/controller/run\"},{\"name\":\"TMPDIR\",\"value\":\"/controller/tmp\"},{\"name\":\"SPOOL_DIRECTORY\",\"value\":\"/controller/wal-restore-spool\"},{\"name\":\"CUSTOM_CNPG_GROUP\",\"value\":\"postgresql.cnpg.io\"},{\"name\":\"CUSTOM_CNPG_VERSION\",\"value\":\"v1\"}],\"resources\":{},\"restartPolicy\":\"Always\",\"volumeMounts\":[{\"name\":\"pgdata\",\"mountPath\":\"/var/lib/postgresql/data\"},{\"name\":\"scratch-data\",\"mountPath\":\"/controller\"},{\"name\":\"shm\",\"mountPath\":\"/dev/shm\"},{\"name\":\"plugins\",\"mountPath\":\"/plugins\"}],\"startupProbe\":{\"exec\":{\"command\":[\"/manager\",\"healthcheck\",\"unix\"]},\"timeoutSeconds\":10,\"failureThreshold\":10},\"securityContext\":{\"capabilities\":{\"drop\":[\"ALL\"]},\"privileged\":false,\"runAsNonRoot\":true,\"readOnlyRootFilesystem\":true,\"allowPrivilegeEscalation\":false,\"seccompProfile\":{\"type\":\"RuntimeDefault\"}}}},{\"op\":\"add\",\"path\":\"/spec/containers/0/volumeMounts/4\",\"value\":{\"name\":\"plugins\",\"mountPath\":\"/plugins\"}}]"}
... removed a ton of of the following lines ...
{"level":"info","ts":"2025-06-03T17:55:25.623490592-05:00","msg":"Pre hook reconciliation completed","name":"postgres-cluster","namespace":"database"}
{"level":"info","ts":"2025-06-03T17:55:26.817776324-05:00","msg":"Pre hook reconciliation start"}
{"level":"debug","ts":"2025-06-03T17:55:26.81791334-05:00","msg":"parsing cluster definition","caller":"/workspace/internal/cnpgi/operator/reconciler.go:61"}
{"level":"debug","ts":"2025-06-03T17:55:26.8182237-05:00","msg":"parsing barman object configuration","name":"postgres-cluster","namespace":"database","caller":"/workspace/internal/cnpgi/operator/reconciler.go:75"}
... removed a ton of the above lines ...
{"level":"info","ts":"2025-06-03T17:58:49.072644351-05:00","msg":"Pre hook reconciliation completed","name":"postgres-cluster","namespace":"database"}
{"level":"info","ts":"2025-06-03T17:58:50.841983113-05:00","logger":"lifecycle","msg":"Lifecycle hook reconciliation start"}
{"level":"info","ts":"2025-06-03T17:58:50.842886488-05:00","logger":"lifecycle","msg":"Reconciling pod"}
{"level":"debug","ts":"2025-06-03T17:58:50.845417656-05:00","logger":"plugin-barman-cloud-lifecycle","msg":"generated patch","podName":"postgres-cluster-1","caller":"/workspace/internal/cnpgi/operator/lifecycle.go:266","content":"[{\"op\":\"replace\",\"path\":\"/spec/initContainers/1/env/8/name\",\"value\":\"TZ\"},{\"op\":\"replace\",\"path\":\"/spec/initContainers/1/env/8/value\",\"value\":\"America/Chicago\"},{\"op\":\"replace\",\"path\":\"/spec/initContainers/1/env/9/name\",\"value\":\"SPOOL_DIRECTORY\"},{\"op\":\"replace\",\"path\":\"/spec/initContainers/1/env/9/value\",\"value\":\"/controller/wal-restore-spool\"},{\"op\":\"replace\",\"path\":\"/spec/initContainers/1/env/10/name\",\"value\":\"CUSTOM_CNPG_GROUP\"},{\"op\":\"replace\",\"path\":\"/spec/initContainers/1/env/10/value\",\"value\":\"postgresql.cnpg.io\"},{\"op\":\"add\",\"path\":\"/spec/initContainers/1/env/11\",\"value\":{\"name\":\"CUSTOM_CNPG_VERSION\",\"value\":\"v1\"}},{\"op\":\"add\",\"path\":\"/spec/initContainers/1/volumeMounts/5\",\"value\":{\"name\":\"k8tz\",\"readOnly\":true,\"mountPath\":\"/usr/share/zoneinfo\"}},{\"op\":\"replace\",\"path\":\"/spec/initContainers/1/startupProbe/periodSeconds\",\"value\":0},{\"op\":\"replace\",\"path\":\"/spec/initContainers/1/startupProbe/successThreshold\",\"value\":0},{\"op\":\"remove\",\"path\":\"/spec/initContainers/1/terminationMessagePath\"},{\"op\":\"remove\",\"path\":\"/spec/initContainers/1/terminationMessagePolicy\"},{\"op\":\"remove\",\"path\":\"/spec/initContainers/1/imagePullPolicy\"}]"}

[Utwo]

The issue still persists on 0.6.0.