From 0bddfb6643acc393efd24d045580592ac89d260b Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Stanislav=20L=C3=A1zni=C4=8Dka?= Date: Mon, 8 Sep 2025 14:39:25 +0200 Subject: [PATCH 1/6] KEP-4192:SVM: remove old goals, move alternative to alternatives MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: Stanislav Láznička --- .../4192-svm-in-tree/README.md | 47 ++++++++++--------- 1 file changed, 26 insertions(+), 21 deletions(-) diff --git a/keps/sig-api-machinery/4192-svm-in-tree/README.md b/keps/sig-api-machinery/4192-svm-in-tree/README.md index 682fbc4150f..c93272552d4 100644 --- a/keps/sig-api-machinery/4192-svm-in-tree/README.md +++ b/keps/sig-api-machinery/4192-svm-in-tree/README.md @@ -94,22 +94,16 @@ This KEP aims to make it easy for users to perform storage migrations without ha - Any modification regarding `StorageVersion` API for HA API servers - Adding logic that relies on the hashed storage versions exposed via the discovery API -### UNCLEAR Goals and/or Non-Goals - -- Automatic storage version migration for CRDs -- Make it easy for Kubernetes developers to drop old API schemas by guaranteeing that storage migration is run automatically on SV hash changes (should this also be on a timer or API server identity?) -- Automated Storage Version Migration via the hash exposed by the `StorageVersion` API - ## Proposal -- Move the existing SVM controller logic in-tree into KCM -- Move the existing SVM REST APIs in-tree (possibly under a new API group to avoid conflicts with the old API being run concurrently) -- For Alpha, we will not trigger automatic storage version migration, and it will be deferred to the user. +- Move the existing SVM controller logic in-tree into KCM from its [original source](https://github.com/kubernetes-sigs/kube-storage-version-migrator). +- Move the existing SVM REST APIs in-tree (possibly under a new API group to avoid conflicts with the old API being run concurrently). +- Automatic storage version migration will be deferred to the user. ### User Stories (Optional) -#### Story 1 **[UNCLEAR]** -As an end user of Kubernetes, we get automatic storage version migration whenever the storage version changes due to an api server upgrade/downgrade. +#### Story 1 +As an end user of Kubernetes, I can trigger storage version migration for my resources. #### Story 2 As an end user using encryption at rest, whenever the key change is detected we can run the storage migration to use the new key for encryption. @@ -250,15 +244,6 @@ When transitioning the Storage Version Migrator in-tree, we will exclusively mov #### Garbage Collection Cache Kube Controller Manager's garbage collection cache contains the name and namespace for all resources, providing a suitable dataset for the migration process. This approach is detailed [here](https://docs.google.com/document/d/1lHDbrMCmNG1KXEpw6gMhDL8qWAWgeSlfW6gbCvD80uw/edit?usp=sharing). _We will use this approach for the Alpha release_. -### Rejected Alternative -#### Streaming List -Currently, the Migrator controller uses the `chunked List` method to retrieve the list of resources from the API server and subsequently perform storage migrations as needed. However, chunked lists are [resource-intensive]((https://github.com/kubernetes/enhancements/blob/master/keps/sig-api-machinery/3157-watch-list/README.md#motivation)) and can lead to a significant overload on the API server, potentially resulting in it being terminated due to out-of-memory (OOM) issues. To address this concern, we have proposed the adoption of an Alpha feature introduced in Kubernetes _v1.27_, known as [`Streaming List`](https://kubernetes.io/docs/reference/using-api/api-concepts/#streaming-lists). - -When the Migrator controller is integrated in-tree, it will leverage the `Streaming List` approach to obtain and monitor resources while executing storage version migrations, as opposed to using the resource-intensive `chunked list` method. In cases where, for any reason, the client cannot establish a streaming watch connection with the API server, it will fall back to the standard `chunked list` method, retaining the older LIST/WATCH semantics. - -- _Open Question_: - - Does `Streaming List` support inconsistent lists? Currently, with chunked lists, we receive inconsistent lists. We need to determine if we can achieve the same with Streaming List. Depending on the outcome, we may consider removing the [ContinueToken](https://github.com/kubernetes-sigs/kube-storage-version-migrator/blob/60dee538334c2366994c2323c0db5db8ab4d2838/pkg/apis/migration/v1alpha1/types.go#L63) from the API. - ### RBAC for SVM - Storage Version Migrator Controller ```yaml @@ -539,7 +524,27 @@ NA ## Alternatives -NA +### Streaming List (considered, rejected) +The original Migrator controller used the _chunked List_ method to retrieve the +list of resources from the API server and subsequently performed storage migrations +as needed. However, chunked lists are [resource-intensive]((https://github.com/kubernetes/enhancements/blob/master/keps/sig-api-machinery/3157-watch-list/README.md#motivation)) +and can lead to a significant overload on the API server, potentially resulting in +it being terminated due to out-of-memory (OOM) issues. To address this concern, we +have proposed the adoption of an Alpha feature introduced in Kubernetes _v1.27_, +known as [`Streaming List`](https://kubernetes.io/docs/reference/using-api/api-concepts/#streaming-lists). + +When the Migrator controller would get integrated in-tree, it would leverage the `Streaming List` +approach to obtain and monitor resources while executing storage version migrations, +as opposed to using the resource-intensive `chunked list` method. In cases where, +for any reason, the client cannot establish a streaming watch connection with the +API server, it will fall back to the standard `chunked list` method, retaining the +older LIST/WATCH semantics. + +- _Open Question for this approach_: + - Does `Streaming List` support inconsistent lists? Currently, with chunked lists, + we receive inconsistent lists. We need to determine if we can achieve the same + with Streaming List. Depending on the outcome, we may consider removing the + [ContinueToken](https://github.com/kubernetes-sigs/kube-storage-version-migrator/blob/60dee538334c2366994c2323c0db5db8ab4d2838/pkg/apis/migration/v1alpha1/types.go#L63) from the API. ## Infrastructure Needed (Optional) From 10d1877e63f02a61fdadbea1aba5ba6a0d62cf22 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Stanislav=20L=C3=A1zni=C4=8Dka?= Date: Mon, 8 Sep 2025 14:45:53 +0200 Subject: [PATCH 2/6] KEP-4192: wrap lines in the design section to minimize future change diffs MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: Stanislav Láznička --- .../sig-api-machinery/4192-svm-in-tree/README.md | 16 +++++++++++++--- 1 file changed, 13 insertions(+), 3 deletions(-) diff --git a/keps/sig-api-machinery/4192-svm-in-tree/README.md b/keps/sig-api-machinery/4192-svm-in-tree/README.md index c93272552d4..0f3eb30b369 100644 --- a/keps/sig-api-machinery/4192-svm-in-tree/README.md +++ b/keps/sig-api-machinery/4192-svm-in-tree/README.md @@ -236,13 +236,23 @@ The final APIs that will be moved in-tree are: ### [Controller](https://github.com/kubernetes-sigs/kube-storage-version-migrator/tree/60dee538334c2366994c2323c0db5db8ab4d2838/pkg/controller) to move #### [Migrator Controller](https://github.com/kubernetes-sigs/kube-storage-version-migrator/tree/60dee538334c2366994c2323c0db5db8ab4d2838/pkg/migrator) -Currently, the Storage Version Migrator comprises two controllers: the `Trigger` controller and the `Migrator` controller. The Trigger controller performs resource discovery, identifying supported resources with the preferred server version every `10 minutes`. Subsequently, the Trigger controller creates the `StorageVersionMigration` resource to initiate the migration process. The Migrator controller then picks up this resource and executes the actual migration. +Currently, the Storage Version Migrator comprises two controllers: the `Trigger` +controller and the `Migrator` controller. The Trigger controller performs resource +discovery, identifying supported resources with the preferred server version every +`10 minutes`. Subsequently, the Trigger controller creates the `StorageVersionMigration` +resource to initiate the migration process. The Migrator controller then picks up this +resource and executes the actual migration. -When transitioning the Storage Version Migrator in-tree, we will exclusively move the Migrator controller as a component of KCM. The creation of the Migration resource will be deferred to the user, instead of being triggered automatically. +When transitioning the Storage Version Migrator in-tree, we will exclusively move the +Migrator controller as a component of KCM. The creation of the Migration resource +will be deferred to the user, instead of being triggered automatically. ### Approach #### Garbage Collection Cache -Kube Controller Manager's garbage collection cache contains the name and namespace for all resources, providing a suitable dataset for the migration process. This approach is detailed [here](https://docs.google.com/document/d/1lHDbrMCmNG1KXEpw6gMhDL8qWAWgeSlfW6gbCvD80uw/edit?usp=sharing). _We will use this approach for the Alpha release_. +Kube Controller Manager's garbage collection cache contains the name and namespace +for all resources, providing a suitable dataset for the migration process. This +approach is detailed [here](https://docs.google.com/document/d/1lHDbrMCmNG1KXEpw6gMhDL8qWAWgeSlfW6gbCvD80uw/edit?usp=sharing). +_We will use this approach for the Alpha release_. ### RBAC for SVM - Storage Version Migrator Controller From ab993774ed27d720ee4953e2b6f323c0fc63098b Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Stanislav=20L=C3=A1zni=C4=8Dka?= Date: Wed, 1 Oct 2025 15:37:46 +0200 Subject: [PATCH 3/6] KEP-4192: minor section style changes MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: Stanislav Láznička --- keps/sig-api-machinery/4192-svm-in-tree/README.md | 11 +++++++---- 1 file changed, 7 insertions(+), 4 deletions(-) diff --git a/keps/sig-api-machinery/4192-svm-in-tree/README.md b/keps/sig-api-machinery/4192-svm-in-tree/README.md index 0f3eb30b369..96b35ff5c06 100644 --- a/keps/sig-api-machinery/4192-svm-in-tree/README.md +++ b/keps/sig-api-machinery/4192-svm-in-tree/README.md @@ -234,9 +234,11 @@ To avoid any conflicts with the Storage Version Migrators running out of tree, w The final APIs that will be moved in-tree are: - `v1alpha1` of `storageversionmigrations.storagemigration.k8s.io` -### [Controller](https://github.com/kubernetes-sigs/kube-storage-version-migrator/tree/60dee538334c2366994c2323c0db5db8ab4d2838/pkg/controller) to move -#### [Migrator Controller](https://github.com/kubernetes-sigs/kube-storage-version-migrator/tree/60dee538334c2366994c2323c0db5db8ab4d2838/pkg/migrator) -Currently, the Storage Version Migrator comprises two controllers: the `Trigger` +## Former, out-of-tree implementation +- [Controller](https://github.com/kubernetes-sigs/kube-storage-version-migrator/tree/60dee538334c2366994c2323c0db5db8ab4d2838/pkg/controller) to move +- [Migrator Controller](https://github.com/kubernetes-sigs/kube-storage-version-migrator/tree/60dee538334c2366994c2323c0db5db8ab4d2838/pkg/migrator) + +Currently, the Storage Version Migrator comprises of two controllers: the `Trigger` controller and the `Migrator` controller. The Trigger controller performs resource discovery, identifying supported resources with the preferred server version every `10 minutes`. Subsequently, the Trigger controller creates the `StorageVersionMigration` @@ -247,7 +249,8 @@ When transitioning the Storage Version Migrator in-tree, we will exclusively mov Migrator controller as a component of KCM. The creation of the Migration resource will be deferred to the user, instead of being triggered automatically. -### Approach +### New KCM-based controller + #### Garbage Collection Cache Kube Controller Manager's garbage collection cache contains the name and namespace for all resources, providing a suitable dataset for the migration process. This From 916232cc7dfa49bc122b60680f48a17d9f67d6d3 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Stanislav=20L=C3=A1zni=C4=8Dka?= Date: Tue, 23 Sep 2025 13:08:37 +0200 Subject: [PATCH 4/6] KEP-4192: update the required permissions MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: Stanislav Láznička --- keps/sig-api-machinery/4192-svm-in-tree/README.md | 9 ++++----- 1 file changed, 4 insertions(+), 5 deletions(-) diff --git a/keps/sig-api-machinery/4192-svm-in-tree/README.md b/keps/sig-api-machinery/4192-svm-in-tree/README.md index 96b35ff5c06..a3c847bf335 100644 --- a/keps/sig-api-machinery/4192-svm-in-tree/README.md +++ b/keps/sig-api-machinery/4192-svm-in-tree/README.md @@ -267,11 +267,10 @@ _We will use this approach for the Alpha release_. rules: - apiGroups: ["*"] resources: ["*"] - verbs: - - get - - list - - watch - - update + verbs: ["patch", "get", "list", "watch"] + - apiGroups: ["storagemigration.k8s.io"] + resources: ["storageversionmigrations/status"] + verbs: ["update"] apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding From cabd637f756b8b8d721de77bb13bdd6d84a179fb Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Stanislav=20L=C3=A1zni=C4=8Dka?= Date: Wed, 1 Oct 2025 15:02:41 +0200 Subject: [PATCH 5/6] KEP-4192: describe the currently implemented SVM behavior MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: Stanislav Láznička --- .../4192-svm-in-tree/README.md | 28 ++++++++++++++++--- 1 file changed, 24 insertions(+), 4 deletions(-) diff --git a/keps/sig-api-machinery/4192-svm-in-tree/README.md b/keps/sig-api-machinery/4192-svm-in-tree/README.md index a3c847bf335..45dc4acf3b4 100644 --- a/keps/sig-api-machinery/4192-svm-in-tree/README.md +++ b/keps/sig-api-machinery/4192-svm-in-tree/README.md @@ -253,9 +253,29 @@ will be deferred to the user, instead of being triggered automatically. #### Garbage Collection Cache Kube Controller Manager's garbage collection cache contains the name and namespace -for all resources, providing a suitable dataset for the migration process. This -approach is detailed [here](https://docs.google.com/document/d/1lHDbrMCmNG1KXEpw6gMhDL8qWAWgeSlfW6gbCvD80uw/edit?usp=sharing). -_We will use this approach for the Alpha release_. +for all resources, providing a suitable dataset for the migration process. + +At the beginning of a migration, to make sure the garbage collector's cache is +up-to-date with the cluster state, the SVM controller performs a list that is +guaranteed to return at most one item (using limits/non-existent namespace names). +The RV of the returned list is then stored and used to compare with the RV from +the GC cache for the given resource. The controller waits until the GC cache RV +is newer that the RV stored for the specific migration. + +The controller then issues patch requests for each of the objects in the GC cache +for the given resource, that is only if the resource version of this object is not greater +than the resource version for the migration that was observed earlier. These patch +requests should contain minimum information: +- API version and kind +- resource name and namespace +- UID from the GC cache + - to make sure we don't accidentally create an empty instance if the resource was + deleted in the meantime +- resource version from the GC cache + - this would provoke conflict errors if the object was modified in the meantime, + meaning we no longer need to attempt the write because someone else already did that + - must also be set in case the object was removed to prevent "immutable field" + errors when setting UID as discussed above ### RBAC for SVM - Storage Version Migrator Controller @@ -393,7 +413,7 @@ total: The feature is enabled using the feature gate `StorageVersionMigrator`. During an upgrade, this gate must be set to true. During a downgrade, this gate must be set to false, and any remaining _StorageVersionMigration_ resources should be manually removed. ### Version Skew Strategy -The feature will be enabled by the feature gate `StorageVersionMigrator` on both the _api-server_ and the _kube-controller-manager_. This gate must be set for both components during installation. Otherwise, since the kube-controller-manager is allowed to be one version lower than the api-server, it won't be able to process any StorageVersionMigration resources created by the API server. +The feature will be enabled by the feature gate `StorageVersionMigrator` on the _kube-controller-manager_. ## Production Readiness Review Questionnaire From e31653990791fe88ebc7aa553e6e524aeefea7d9 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Stanislav=20L=C3=A1zni=C4=8Dka?= Date: Wed, 1 Oct 2025 15:48:41 +0200 Subject: [PATCH 6/6] KEP-4192: update ToC MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: Stanislav Láznička --- keps/sig-api-machinery/4192-svm-in-tree/README.md | 11 ++++------- 1 file changed, 4 insertions(+), 7 deletions(-) diff --git a/keps/sig-api-machinery/4192-svm-in-tree/README.md b/keps/sig-api-machinery/4192-svm-in-tree/README.md index 45dc4acf3b4..1f4950c7cf2 100644 --- a/keps/sig-api-machinery/4192-svm-in-tree/README.md +++ b/keps/sig-api-machinery/4192-svm-in-tree/README.md @@ -6,10 +6,9 @@ - [Motivation](#motivation) - [Goals](#goals) - [Non-Goals](#non-goals) - - [UNCLEAR Goals and/or Non-Goals](#unclear-goals-andor-non-goals) - [Proposal](#proposal) - [User Stories (Optional)](#user-stories-optional) - - [Story 1 [UNCLEAR]](#story-1-unclear) + - [Story 1](#story-1) - [Story 2](#story-2) - [Notes/Constraints/Caveats (Optional)](#notesconstraintscaveats-optional) - [Risks and Mitigations](#risks-and-mitigations) @@ -17,12 +16,9 @@ - [APIs to move](#apis-to-move) - [We will move following APIs in-tree:](#we-will-move-following-apis-in-tree) - [Changes while we move above APIs in-tree:](#changes-while-we-move-above-apis-in-tree) - - [Controller to move](#controller-to-move) - - [Migrator Controller](#migrator-controller) - - [Approach](#approach) +- [Former, out-of-tree implementation](#former-out-of-tree-implementation) + - [New KCM-based controller](#new-kcm-based-controller) - [Garbage Collection Cache](#garbage-collection-cache) - - [Rejected Alternative](#rejected-alternative) - - [Streaming List](#streaming-list) - [RBAC for SVM](#rbac-for-svm) - [Test Plan](#test-plan) - [Prerequisite testing updates](#prerequisite-testing-updates) @@ -44,6 +40,7 @@ - [Implementation History](#implementation-history) - [Drawbacks](#drawbacks) - [Alternatives](#alternatives) + - [Streaming List (considered, rejected)](#streaming-list-considered-rejected) - [Infrastructure Needed (Optional)](#infrastructure-needed-optional)