[DIFF] kcp-1.33.3

KCP_RELATED_CHANGES.md

@@ -0,0 +1,97 @@
+# Why this forked repository ?
+
+This repository carries the prototype branch which accumulates the hacks, prototypes, proto-KEP experiments, and workarounds required to make [KCP](https://github.com/kcp-dev/kcp/blob/main/README.md) a reality.
+It is based on K8S 1.22 for now and commits are identified with basic labels like HACK/FEATURE/WORKAROUND.
+
+# Summary of changes
+
+The detailed explanation of the changes made on top of the Kubernetes code can be found in both the commit messages, and comments of the associated code.
+
+However here is a summary of the changes, with the underlying requirements and motivations. Reading the linked investigation document first will help.
+
+## A. Minimal API Server
+
+__Investigation document:__ [minimal-api-server.md](https://github.com/kcp-dev/kcp/blob/main/docs/investigations/minimal-api-server.md)
+
+1.  New generic control plane based on kube api-server
+
+    It is mainly provided by code:
+
+    1. initially duplicated from kube api-server main code, and legacy scheme (`DUPLICATE` commits),
+
+    2. then stripped down from unnecessary things (ergress, api aggregation, webhooks) and APIs (Pods, Nodes, Deployments, etc ...) (`NEW` commits)
+
+2.  Support adding K8S built-in resources (`core/Pod`, `apps/Deployment`, ...) as CRDs
+
+    This is required since the new generic control plane scheme doesn't contain those resources any more.
+
+    This is provided by:
+
+    - hacks (`HACK` commits) that:
+
+      1. <a id="A-2-1"></a> allow the go-restful server to be bypassed for those resources, and route them to the CRD handler
+      2. <a id="A-2-2"></a> allow the CRD handler, and opanapi publisher, to support resources of the `core` group
+      3. <a id="A-2-3"></a> convert the `protobuf` requests sent to those resources resources to requests in the `application/json` content type, before letting the CRD handler serve them
+      4. <a id="A-2-4"></a> replace the table converter of CRDs that bring back those resources, with the default table converter of the related built-in resource
+
+    - a new feature, or potential kube fix (`KUBEFIX` commit), that:
+
+      5. <a id="A-2-5"></a> introduces the support of strategic merge patch for CRDs.
+          This support uses the OpenAPI v3 schema of the CRD to drive the SMP execution, but only adds a minimal implementation and doesn't fully support OpenAPI schemas that don't have expected `patchStrategy` and `patchMergeKey` annotations.
+          In order to avoid changing the behavior of existing client tools, the support is only added for those K8S built-in resources 
+
+## B. Logical clusters
+
+__Investigation document:__ [logical-clusters.md](https://github.com/kcp-dev/kcp/blob/main/docs/investigations/logical-clusters.md)
+
+1.  Logical clusters represented as a prefix in etcd
+
+    It is mainly provided by hacks (`HACK` commits) that:
+
+    1. <a id="B-1-1"></a> allow intercepting the api server handler chain to set the expected logical cluster context value from either a given suffix in the request APIServer base URL, or a given header in the http request
+
+    2. <a id="B-1-2"></a> change etcd storage layer in order to use the logical cluster as a prefix in the etcd key
+
+    3. <a id="B-1-3"></a> allow wildcard watches that retrieve objects from all the logical clusters
+
+    4. <a id="B-1-4"></a> correctly get or set the `clusterName` metadata field in the storage layer operations based on the etcd key and its new prefix
+
+2.  Support of logical clusters (== tenancy) in the CRD management, OpenAPI and discovery endpoints, and clients used by controllers
+
+    <a id="B-2"></a>It is mainly provided by a hack (`HACK` commit) that adds CRD tenancy by ensuring that logical clusters are taken in account in:
+    - CRD-related controllers
+    - APIServices-related controllers
+    - Discovery + OpenAPI endpoints
+
+    In the current Kubernetes design, those 3 areas are highly coupled and intricated, which explains why this commit had to hack the code at various levels:
+    - client-go level
+    - controllers level,
+    - http handlers level.
+
+    While this gives a detailed idea of which code needs to be touched in order to enable CRD tenancy, a clean implementation would first require some refactoring, in order to build the required abstraction layers that would allow decoupling those areas.
+
+# Potential client problems
+
+Although these changes in the K8S codebase were made in order to keep the compatibility with Kuberntes client tools, there might be some problems:
+
+## Incomplete protobuf support for built-in resources
+
+In some contexts, like the `controller-runtime` library used by the Operator SDK, all the resources of the `client-go` scheme are created / updated using the `application/vnd.kubernetes.protobuf` content type.
+
+However when these resources are in fact added as CRDs, in the KCP minimal API server scenario, these resources cannot be created / updated since the protobuf (de)serialization is not (and probably cannot be) supported for CRDs.
+So for now in this case, the [A.2.3 hack mentioned above](#A-2-3) just converts the `protobuf` request to a `json` one, but this might not cover all the use-cases or corner cases.
+
+The clean solution would probably be the negotiation of serialization type in `client-go`, which we haven't implemented yet, but which would work like this:
+When a request for an unsupported serialization is returned, the server should reject it with a 406
+and provide a list of supported content types. `client-go` should then examine whether it can satisfy such a request by encoding the object with a different scheme.
+This would require a KEP but at least is in keeping with content negotiation on GET / WATCH in Kube
+
+## Incomplete Strategic merge patch support for built-in resources
+
+Client tools like `kubectl` assume that all K8S native resources (== `client-go` schema resources)
+support strategic merge patch, and use it by default when updating or patching a resource.
+
+In Kube, currently, strategic merge patch is not supported for CRDs, which would break compatibility with client tools for all the K8S natives resources that are in fact added as CRD in the KCP minimal api server.
+The [A-2-5 change mentioned above](#A-2-5) tries to fix this by using the CRD openAPI v3 schema as the source of the required information that will drive the strategic merge patch execution.
+
+While this fixes the problem in most cases, there might still be errors in case the OpenAPI v2 schema for such a resource is missing `x-kubernetes-patch-strategy` and `x-kubernetes-patch-merge-key` annotations when imported from the CRD OpenAPI v3 schema.

cmd/kube-apiserver/app/config.go

@@ -18,7 +18,9 @@ package app
 
 import (
 	apiextensionsapiserver "k8s.io/apiextensions-apiserver/pkg/apiserver"
+	"k8s.io/apiextensions-apiserver/pkg/apiserver/conversion"
 	"k8s.io/apimachinery/pkg/runtime"
+	"k8s.io/apiserver/pkg/informerfactoryhack"
 	"k8s.io/apiserver/pkg/util/webhook"
 	aggregatorapiserver "k8s.io/kube-aggregator/pkg/apiserver"
 	aggregatorscheme "k8s.io/kube-aggregator/pkg/apiserver/scheme"
@@ -92,14 +94,23 @@ func NewConfig(opts options.CompletedOptions) (*Config, error) {
 	}
 	c.KubeAPIs = kubeAPIs
 
-	apiExtensions, err := controlplaneapiserver.CreateAPIExtensionsConfig(*kubeAPIs.ControlPlane.Generic, kubeAPIs.ControlPlane.VersionedInformers, pluginInitializer, opts.CompletedOptions, opts.MasterCount,
-		serviceResolver, webhook.NewDefaultAuthenticationInfoResolverWrapper(kubeAPIs.ControlPlane.ProxyTransport, kubeAPIs.ControlPlane.Generic.EgressSelector, kubeAPIs.ControlPlane.Generic.LoopbackClientConfig, kubeAPIs.ControlPlane.Generic.TracerProvider))
+	authInfoResolver := webhook.NewDefaultAuthenticationInfoResolverWrapper(kubeAPIs.ControlPlane.ProxyTransport, kubeAPIs.ControlPlane.Generic.EgressSelector, kubeAPIs.ControlPlane.Generic.LoopbackClientConfig, kubeAPIs.ControlPlane.Generic.TracerProvider)
+	conversionFactory, err := conversion.NewCRConverterFactory(serviceResolver, authInfoResolver)
+	if err != nil {
+		return nil, err
+	}
+
+	// TODO(ntnn): upstream uses kubeAPIs.ControlPlane.VersionedInformers instead of the versionedInformers returned by BuildGenericConfig.
+	// Check if these are equivalent or if this is a deliberate
+	// diversion.
+	apiExtensions, err := controlplaneapiserver.CreateAPIExtensionsConfig(*kubeAPIs.ControlPlane.Generic, informerfactoryhack.Wrap(versionedInformers), pluginInitializer, opts.CompletedOptions, opts.MasterCount, conversionFactory)
 	if err != nil {
 		return nil, err
 	}
 	c.ApiExtensions = apiExtensions
 
-	aggregator, err := controlplaneapiserver.CreateAggregatorConfig(*kubeAPIs.ControlPlane.Generic, opts.CompletedOptions, kubeAPIs.ControlPlane.VersionedInformers, serviceResolver, kubeAPIs.ControlPlane.ProxyTransport, kubeAPIs.ControlPlane.Extra.PeerProxy, pluginInitializer)
+	// TODO(ntnn): Here as well.
+	aggregator, err := controlplaneapiserver.CreateAggregatorConfig(*kubeAPIs.ControlPlane.Generic, opts.CompletedOptions, informerfactoryhack.Wrap(versionedInformers), serviceResolver, kubeAPIs.ControlPlane.ProxyTransport, kubeAPIs.ControlPlane.Extra.PeerProxy, pluginInitializer)
 	if err != nil {
 		return nil, err
 	}

cmd/kube-apiserver/app/server.go

@@ -25,12 +25,14 @@ import (
 	"net/url"
 	"os"
 
+	kcpinformers "github.com/kcp-dev/client-go/informers"
 	"github.com/spf13/cobra"
 	apiextensionsv1 "k8s.io/apiextensions-apiserver/pkg/apis/apiextensions/v1"
 	utilerrors "k8s.io/apimachinery/pkg/util/errors"
 	utilruntime "k8s.io/apimachinery/pkg/util/runtime"
 	"k8s.io/apiserver/pkg/admission"
 	genericapifilters "k8s.io/apiserver/pkg/endpoints/filters"
+	"k8s.io/apiserver/pkg/informerfactoryhack"
 	genericapiserver "k8s.io/apiserver/pkg/server"
 	"k8s.io/apiserver/pkg/server/egressselector"
 	serverstorage "k8s.io/apiserver/pkg/server/storage"
@@ -184,7 +186,7 @@ func CreateServerChain(config CompletedConfig) (*aggregatorapiserver.APIAggregat
 	}
 
 	// aggregator comes last in the chain
-	aggregatorServer, err := controlplaneapiserver.CreateAggregatorServer(config.Aggregator, kubeAPIServer.ControlPlane.GenericAPIServer, apiExtensionsServer.Informers.Apiextensions().V1().CustomResourceDefinitions(), crdAPIEnabled, apiVersionPriorities)
+	aggregatorServer, err := controlplaneapiserver.CreateAggregatorServer(config.Aggregator, kubeAPIServer.ControlPlane.GenericAPIServer, apiExtensionsServer.Informers.Apiextensions().V1().CustomResourceDefinitions().Cluster(controlplaneapiserver.LocalAdminCluster), crdAPIEnabled, apiVersionPriorities)
 	if err != nil {
 		// we don't need special handling for innerStopCh because the aggregator server doesn't create any go routines
 		return nil, err
@@ -197,7 +199,7 @@ func CreateServerChain(config CompletedConfig) (*aggregatorapiserver.APIAggregat
 func CreateKubeAPIServerConfig(
 	opts options.CompletedOptions,
 	genericConfig *genericapiserver.Config,
-	versionedInformers clientgoinformers.SharedInformerFactory,
+	versionedInformers kcpinformers.SharedInformerFactory,
 	storageFactory *serverstorage.DefaultStorageFactory,
 ) (
 	*controlplane.Config,
@@ -215,7 +217,7 @@ func CreateKubeAPIServerConfig(
 		return nil, nil, nil, fmt.Errorf("failed to create admission plugin initializer: %w", err)
 	}
 
-	serviceResolver := buildServiceResolver(opts.EnableAggregatorRouting, genericConfig.LoopbackClientConfig.Host, versionedInformers)
+	serviceResolver := buildServiceResolver(opts.EnableAggregatorRouting, genericConfig.LoopbackClientConfig.Host, informerfactoryhack.Wrap(versionedInformers))
 	controlplaneConfig, admissionInitializers, err := controlplaneapiserver.CreateConfig(opts.CompletedOptions, genericConfig, versionedInformers, storageFactory, serviceResolver, kubeInitializers)
 	if err != nil {
 		return nil, nil, nil, err

cmd/kube-controller-manager/app/core.go

@@ -28,6 +28,8 @@ import (
 	"time"
 
 	v1 "k8s.io/api/core/v1"
+	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
+	"k8s.io/apimachinery/pkg/runtime/schema"
 	genericfeatures "k8s.io/apiserver/pkg/features"
 	"k8s.io/apiserver/pkg/quota/v1/generic"
 	utilfeature "k8s.io/apiserver/pkg/util/feature"
@@ -577,7 +579,9 @@ func startModifiedNamespaceController(ctx context.Context, controllerContext Con
 		return nil, true, err
 	}
 
-	discoverResourcesFn := namespaceKubeClient.Discovery().ServerPreferredNamespacedResources
+	discoverResourcesFn := func(clusterName logicalcluster.Name) ([]*metav1.APIResourceList, error) {
+		return namespaceKubeClient.Discovery().ServerPreferredNamespacedResources()
+	}
 
 	namespaceController := namespacecontroller.NewNamespaceController(
 		ctx,

hack/kcp/garbage_collector_patch.go

@@ -0,0 +1,109 @@
+/*
+Copyright 2022 The KCP Authors.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
+package main
+
+import (
+	"bufio"
+	"bytes"
+	"fmt"
+	"go/ast"
+	"go/format"
+	"go/parser"
+	"go/token"
+	"log"
+	"strings"
+)
+
+/*
+Process:
+
+1. go run ./hack/kcp/garbage_collector_patch.go > pkg/controller/garbagecollector/garbagecollector_kcp.go
+(you may need to add -mod=readonly)
+
+2. goimports -w pkg/controller/garbagecollector/garbagecollector_kcp.go
+
+3. reapply patch for kcp to pkg/controller/garbagecollector/garbagecollector_kcp.go
+*/
+
+func main() {
+	fileSet := token.NewFileSet()
+
+	file, err := parser.ParseFile(fileSet, "pkg/controller/garbagecollector/garbagecollector.go", nil, parser.ParseComments)
+	if err != nil {
+		log.Fatal(err)
+	}
+
+	// n stores a reference to the node for the function declaration for Sync
+	var n ast.Node
+
+	ast.Inspect(file, func(node ast.Node) bool {
+		switch x := node.(type) {
+		case *ast.FuncDecl:
+			if x.Name.Name == "Sync" {
+				// Store the reference
+				n = node
+				// Stop further inspection
+				return false
+			}
+		}
+
+		// Continue recursing
+		return true
+	})
+
+	startLine := fileSet.Position(n.Pos()).Line
+	endLine := fileSet.Position(n.End()).Line
+
+	// To preserve the comments from within the function body itself, we have to write out the entire file to a buffer,
+	// then extract only the lines we care about (the function body).
+	var buf bytes.Buffer
+	if err := format.Node(&buf, fileSet, file); err != nil {
+		log.Fatal(err)
+	}
+
+	// Convert the buffer to a slice of lines, so we can grab the portion we want
+	var lines []string
+	scanner := bufio.NewScanner(&buf)
+	for scanner.Scan() {
+		lines = append(lines, scanner.Text())
+	}
+	if err := scanner.Err(); err != nil {
+		log.Fatal(err)
+	}
+
+	fmt.Println(`/*
+Copyright 2022 The KCP Authors.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
+package garbagecollector
+`)
+
+	// Finally, print the line range we need
+	fmt.Println(strings.Join(lines[startLine-1:endLine], "\n"))
+}