Disable StorageNamespaceIndex feature gate when BtreeWatchCache is enabled and deprecate it

Previously, the cache used a map keyed by the full object key,
requiring iteration and filtering by namespace for namespace-scoped requests.
This index allowed for faster responses by avoiding this iteration.

With the introduction of the BtreeWatchCache, this optimization is no longer necessary.
The B-tree structure allows efficient prefix-based searches,
including fetching objects by namespace.
Furthermore, the B-tree returns elements ordered by key, eliminating the need for separate sorting.

Performance improvements with the BtreeWatchCache have been validated through benchmarks matching K8s scalability dimentions (see table below).
These results demonstrate that the B-tree approach provides comparable or better performance than the map with index.
Therefore, the StorageNamespaceIndex feature flag can be safely flipped to false and subsequently deprecated.

| Benchmark                                                                         | Btree with Index (current) | Btree without Index    | Map with Index         | Map without Index (sanity check) |
| --------------------------------------------------------------------------------- | -------------------------- | ---------------------- | ---------------------- | -------------------------------- |
| StoreList (10k Namespaces, 150k Pods, 5k Nodes, RV=, Namespace Scope)             | 20.77µs ± 10%              | 20.14µs ± 13% (~0%)    | 19.73µs ± 6% (~0%)     | 1067.34µs ± 10% (+5037.73%)      |
| StoreList (10k Namespaces, 150k Pods, 5k Nodes, RV=NotOlderThan, Namespace Scope) | 3.943µs ± 6%               | 3.928µs ± 6% (~0%)     | 3.665µs ± 3% (-7.05%)  | 944.641µs ± 1% (+23857.41%)      |
| StoreList (50 Namespaces, 150k Pods, 5k Nodes, RV=, Namespace Scope)              | 303.3µs ± 2%               | 258.2µs ± 2% (-14.85%) | 340.1µs ± 3% (+12.15%) | 1668.6µs ± 4% (+450.23%)         |
| StoreList (50 Namespaces, 150k Pods, 5k Nodes, RV=NotOlderThan, Namespace Scope)  | 286.2µs ± 3%               | 234.7µs ± 1% (-17.99%) | 326.9µs ± 2% (+14.22%) | 1347.7µs ± 4% (+370.91%)         |
| StoreList (100 Namespaces, 110k Pods, 1k Nodes, RV=, Namespace Scope)             | 125.3µs ± 2%               | 112.3µs ± 5% (-10.38%) | 137.5µs ± 2% (+9.81%)  | 1395.1µs ± 8% (+1013.78%)        |
| StoreList (100 Namespaces, 110k Pods, 1k Nodes, RV=NotOlderThan, Namespace Scope) | 120.6µs ± 2%               | 113.2µs ± 1% (-6.13%)  | 133.8µs ± 1% (+10.92%) | 1719.1µs ± 5% (+1325.35%)        |
| Geometric Mean                                                                    | 68.94µs                    | 62.73µs (-9.02%)       | 72.72µs (+5.48%)       | 1.326ms (+1823.40%)              |

Author: serathius

pkg/features/kube_features.go

@@ -681,6 +681,7 @@ const (
 	// owner: @ahutsunshine
 	//
 	// Allows namespace indexer for namespace scope resources in apiserver cache to accelerate list operations.
+	// Superseded by BtreeWatchCache.
 	StorageNamespaceIndex featuregate.Feature = "StorageNamespaceIndex"
 
 	// owner: @nilekhc

pkg/features/versioned_kube_features.go

@@ -675,6 +675,7 @@ var defaultVersionedKubernetesFeatureGates = map[featuregate.Feature]featuregate
 
 	StorageNamespaceIndex: {
 		{Version: version.MustParse("1.30"), Default: true, PreRelease: featuregate.Beta},
+		{Version: version.MustParse("1.33"), Default: true, PreRelease: featuregate.Deprecated},
 	},
 
 	ServiceAccountNodeAudienceRestriction: {

pkg/registry/core/pod/strategy.go

@@ -40,6 +40,7 @@ import (
 	utilnet "k8s.io/apimachinery/pkg/util/net"
 	utilvalidation "k8s.io/apimachinery/pkg/util/validation"
 	"k8s.io/apimachinery/pkg/util/validation/field"
+	apiserverfeatures "k8s.io/apiserver/pkg/features"
 	"k8s.io/apiserver/pkg/registry/generic"
 	"k8s.io/apiserver/pkg/storage"
 	"k8s.io/apiserver/pkg/storage/names"
@@ -365,7 +366,7 @@ func GetAttrs(obj runtime.Object) (labels.Set, fields.Set, error) {
 // MatchPod returns a generic matcher for a given label and field selector.
 func MatchPod(label labels.Selector, field fields.Selector) storage.SelectionPredicate {
 	var indexFields = []string{"spec.nodeName"}
-	if utilfeature.DefaultFeatureGate.Enabled(features.StorageNamespaceIndex) {
+	if utilfeature.DefaultFeatureGate.Enabled(features.StorageNamespaceIndex) && !utilfeature.DefaultFeatureGate.Enabled(apiserverfeatures.BtreeWatchCache) {
 		indexFields = append(indexFields, "metadata.namespace")
 	}
 	return storage.SelectionPredicate{
@@ -404,7 +405,7 @@ func Indexers() *cache.Indexers {
 	var indexers = cache.Indexers{
 		storage.FieldIndex("spec.nodeName"): NodeNameIndexFunc,
 	}
-	if utilfeature.DefaultFeatureGate.Enabled(features.StorageNamespaceIndex) {
+	if utilfeature.DefaultFeatureGate.Enabled(features.StorageNamespaceIndex) && !utilfeature.DefaultFeatureGate.Enabled(apiserverfeatures.BtreeWatchCache) {
 		indexers[storage.FieldIndex("metadata.namespace")] = NamespaceIndexFunc
 	}
 	return &indexers

test/featuregates_linter/test_data/versioned_feature_list.yaml

@@ -1286,6 +1286,10 @@
     lockToDefault: false
     preRelease: Beta
     version: "1.30"
+  - default: true
+    lockToDefault: false
+    preRelease: Deprecated
+    version: "1.33"
 - name: StorageVersionAPI
   versionedSpecs:
   - default: false