Linkerd 2.12.0 failing to create applicable manifest with inject

[amartincolby]

What is the issue?

I am running Microk8s which uses Linkerd v2.12.0. I am trying to get the deployments saved as a file, injecting into the file, then applying them again with kubectl. This is failing. For example, I get the error "Operation cannot be fulfilled on statefulsets.apps "keycloak-postgresql": the object has been modified; please apply your changes to the latest version and try again."

How can it be reproduced?

Run Microk8s (or theoretically any K8 distro), and try to inject Linkerd via a file, then applying it to the cluster.

Logs, error output, etc

Error from server (Conflict): error when applying patch:
{"metadata":{"annotations":{"kubectl.kubernetes.io/last-applied-configuration":"{\"apiVersion\":\"apps/v1\",\"kind\":\"StatefulSet\",\"metadata\":{\"annotations\":{\"meta.helm.sh/release-name\":\"keycloak\",\"meta.helm.sh/release-namespace\":\"default\"},\"generation\":3,\"labels\":{\"app.kubernetes.io/component\":\"primary\",\"app.kubernetes.io/instance\":\"keycloak\",\"app.kubernetes.io/managed-by\":\"Helm\",\"app.kubernetes.io/name\":\"postgresql\",\"helm.sh/chart\":\"postgresql-12.1.2\"},\"name\":\"keycloak-postgresql\",\"namespace\":\"default\",\"resourceVersion\":\"368356\",\"uid\":\"1ab8f527-4b55-434b-bfd6-39e68b167f4b\"},\"spec\":{\"podManagementPolicy\":\"OrderedReady\",\"replicas\":1,\"revisionHistoryLimit\":10,\"selector\":{\"matchLabels\":{\"app.kubernetes.io/component\":\"primary\",\"app.kubernetes.io/instance\":\"keycloak\",\"app.kubernetes.io/name\":\"postgresql\"}},\"serviceName\":\"keycloak-postgresql-hl\",\"template\":{\"metadata\":{\"annotations\":{\"linkerd.io/inject\":\"enabled\"},\"labels\":{\"app.kubernetes.io/component\":\"primary\",\"app.kubernetes.io/instance\":\"keycloak\",\"app.kubernetes.io/managed-by\":\"Helm\",\"app.kubernetes.io/name\":\"postgresql\",\"helm.sh/chart\":\"postgresql-12.1.2\"},\"name\":\"keycloak-postgresql\"},\"spec\":{\"affinity\":{\"podAntiAffinity\":{\"preferredDuringSchedulingIgnoredDuringExecution\":[{\"podAffinityTerm\":{\"labelSelector\":{\"matchLabels\":{\"app.kubernetes.io/component\":\"primary\",\"app.kubernetes.io/instance\":\"keycloak\",\"app.kubernetes.io/name\":\"postgresql\"}},\"topologyKey\":\"kubernetes.io/hostname\"},\"weight\":1}]}},\"containers\":[{\"env\":[{\"name\":\"BITNAMI_DEBUG\",\"value\":\"false\"},{\"name\":\"POSTGRESQL_PORT_NUMBER\",\"value\":\"5432\"},{\"name\":\"POSTGRESQL_VOLUME_DIR\",\"value\":\"/bitnami/postgresql\"},{\"name\":\"PGDATA\",\"value\":\"/bitnami/postgresql/data\"},{\"name\":\"POSTGRES_USER\",\"value\":\"bn_keycloak\"},{\"name\":\"POSTGRES_POSTGRES_PASSWORD\",\"valueFrom\":{\"secretKeyRef\":{\"key\":\"postgres-password\",\"name\":\"keycloak-postgresql\"}}},{\"name\":\"POSTGRES_PASSWORD\",\"valueFrom\":{\"secretKeyRef\":{\"key\":\"password\",\"name\":\"keycloak-postgresql\"}}},{\"name\":\"POSTGRES_DB\",\"value\":\"bitnami_keycloak\"},{\"name\":\"POSTGRESQL_ENABLE_LDAP\",\"value\":\"no\"},{\"name\":\"POSTGRESQL_ENABLE_TLS\",\"value\":\"no\"},{\"name\":\"POSTGRESQL_LOG_HOSTNAME\",\"value\":\"false\"},{\"name\":\"POSTGRESQL_LOG_CONNECTIONS\",\"value\":\"false\"},{\"name\":\"POSTGRESQL_LOG_DISCONNECTIONS\",\"value\":\"false\"},{\"name\":\"POSTGRESQL_PGAUDIT_LOG_CATALOG\",\"value\":\"off\"},{\"name\":\"POSTGRESQL_CLIENT_MIN_MESSAGES\",\"value\":\"error\"},{\"name\":\"POSTGRESQL_SHARED_PRELOAD_LIBRARIES\",\"value\":\"pgaudit\"}],\"image\":\"docker.io/bitnami/postgresql:15.1.0-debian-11-r0\",\"imagePullPolicy\":\"IfNotPresent\",\"livenessProbe\":{\"exec\":{\"command\":[\"/bin/sh\",\"-c\",\"exec pg_isready -U \\\"bn_keycloak\\\" -d \\\"dbname=bitnami_keycloak\\\" -h 127.0.0.1 -p 5432\"]},\"failureThreshold\":6,\"initialDelaySeconds\":30,\"periodSeconds\":10,\"successThreshold\":1,\"timeoutSeconds\":5},\"name\":\"postgresql\",\"ports\":[{\"containerPort\":5432,\"name\":\"tcp-postgresql\",\"protocol\":\"TCP\"}],\"readinessProbe\":{\"exec\":{\"command\":[\"/bin/sh\",\"-c\",\"-e\",\"exec pg_isready -U \\\"bn_keycloak\\\" -d \\\"dbname=bitnami_keycloak\\\" -h 127.0.0.1 -p 5432\\n[ -f /opt/bitnami/postgresql/tmp/.initialized ] || [ -f /bitnami/postgresql/.initialized ]\\n\"]},\"failureThreshold\":6,\"initialDelaySeconds\":5,\"periodSeconds\":10,\"successThreshold\":1,\"timeoutSeconds\":5},\"resources\":{\"requests\":{\"cpu\":\"250m\",\"memory\":\"256Mi\"}},\"securityContext\":{\"runAsUser\":1001},\"terminationMessagePath\":\"/dev/termination-log\",\"terminationMessagePolicy\":\"File\",\"volumeMounts\":[{\"mountPath\":\"/dev/shm\",\"name\":\"dshm\"},{\"mountPath\":\"/bitnami/postgresql\",\"name\":\"data\"}]}],\"dnsPolicy\":\"ClusterFirst\",\"restartPolicy\":\"Always\",\"schedulerName\":\"default-scheduler\",\"securityContext\":{\"fsGroup\":1001},\"serviceAccount\":\"default\",\"serviceAccountName\":\"default\",\"terminationGracePeriodSeconds\":30,\"volumes\":[{\"emptyDir\":{\"medium\":\"Memory\"},\"name\":\"dshm\"}]}},\"updateStrategy\":{\"rollingUpdate\":{\"partition\":0},\"type\":\"RollingUpdate\"},\"volumeClaimTemplates\":[{\"apiVersion\":\"v1\",\"kind\":\"PersistentVolumeClaim\",\"metadata\":{\"name\":\"data\"},\"spec\":{\"accessModes\":[\"ReadWriteOnce\"],\"resources\":{\"requests\":{\"storage\":\"8Gi\"}},\"volumeMode\":\"Filesystem\"},\"status\":{\"phase\":\"Pending\"}}]},\"status\":{\"availableReplicas\":0,\"collisionCount\":0,\"currentRevision\":\"keycloak-postgresql-777bff885b\",\"observedGeneration\":3,\"replicas\":1,\"updateRevision\":\"keycloak-postgresql-58d8579c4\",\"updatedReplicas\":1}}\n"},"resourceVersion":"368356"},"spec":{"template":{"metadata":{"annotations":{"linkerd.io/inject":"enabled"}}},"volumeClaimTemplates":[{"apiVersion":"v1","kind":"PersistentVolumeClaim","metadata":{"name":"data"},"spec":{"accessModes":["ReadWriteOnce"],"resources":{"requests":{"storage":"8Gi"}},"volumeMode":"Filesystem"},"status":{"phase":"Pending"}}]},"status":{"availableReplicas":0,"currentReplicas":null,"currentRevision":"keycloak-postgresql-777bff885b"}}
to:
Resource: "apps/v1, Resource=statefulsets", GroupVersionKind: "apps/v1, Kind=StatefulSet"
Name: "keycloak-postgresql", Namespace: "default"
for: "./linkerd_injected_config_keycloak.yml": error when patching "./linkerd_injected_config_keycloak.yml": Operation cannot be fulfilled on statefulsets.apps "keycloak-postgresql": the object has been modified; please apply your changes to the latest version and try again

output of linkerd check -o short

Linkerd core checks

linkerd-version

‼ cli is up-to-date
is running version 2.12.0 but the latest stable version is 2.12.4
see https://linkerd.io/2.12/checks/#l5d-version-cli for hints

control-plane-version

‼ control plane is up-to-date
is running version 2.12.0 but the latest stable version is 2.12.4
see https://linkerd.io/2.12/checks/#l5d-version-control for hints

linkerd-control-plane-proxy

‼ control plane proxies are up-to-date
some proxies are not running the current version:
* linkerd-identity-695f758459-x2z7g (stable-2.12.0)
* linkerd-destination-cddf9bc7-w72tb (stable-2.12.0)
* linkerd-proxy-injector-56d67f9dbf-l97pg (stable-2.12.0)
see https://linkerd.io/2.12/checks/#l5d-cp-proxy-version for hints

Linkerd extensions checks

linkerd-viz

‼ viz extension proxies are up-to-date
some proxies are not running the current version:
* web-849d654c75-8cp8d (stable-2.12.0)
* metrics-api-6dd748b8c7-c4t8l (stable-2.12.0)
* tap-846b668b95-rx7pv (stable-2.12.0)
* tap-injector-86d66f779c-8tnjb (stable-2.12.0)
* prometheus-7689669f9b-lwmcd (stable-2.12.0)
see https://linkerd.io/2.12/checks/#l5d-viz-proxy-cp-version for hints

Status check results are √

Environment

Microk8s 1.26.1
Windows 11
Linkerd 2.12.0

Possible solution

Upgrading my Linkerd version. I just want to confirm that will do it before requesting an upgrade in the Microk8s project.

If I remove resourceVersion in the manifest file, I can then successfully apply the file.

Additional context

No response

Would you like to work on fixing this bug?

None

[kleimkuhler]

This is likely an issue with the files are being created, rather than injecting Linkerd related. The error you're seeing Operation cannot be fulfilled on ...: the object has been modified is shown when trying to apply manifests that have autogenerated fields such as creationTimestamp, resourceVersion, etc. I'd make sure to remove the autogenerated fields from the manifests before trying to apply them to the cluster.

[amartincolby]

This was not a problem earlier. The only change to my stack has been Linkerd, which I updated. Is not Linkerd supposed to sanitize files when injecting?

[amartincolby]

Nevermind. I was just able to replicate with version 2.11.1, which was my previous version. I still don't know what is happening. I think it may be because I updated to the newest PowerShell a few days ago.

[amartincolby]

Yep. I am able to fix it by removing the resourceVersion. I removed all other fields and it is absolutely the only one causing the issue. I thought that Linkerd, when injecting into a manifest file, sanitized the file.