Merge pull request kubernetes#4667 from pohly/dra-kubelet-no-api-dependency

KEP-4318: DRA: avoid kubelet API version dependency

Author: k8s-ci-robot

keps/sig-node/4381-dra-structured-parameters/README.md

@@ -78,7 +78,7 @@ SIG Architecture for cross-cutting KEPs).
   - [Risks and Mitigations](#risks-and-mitigations)
     - [Feature not used](#feature-not-used)
     - [Compromised node](#compromised-node)
-    - [Compromised resource driver plugin](#compromised-resource-driver-plugin)
+    - [Compromised kubelet plugin](#compromised-kubelet-plugin)
     - [User permissions and quotas](#user-permissions-and-quotas)
     - [Usability](#usability)
 - [Design Details](#design-details)
@@ -110,9 +110,10 @@ SIG Architecture for cross-cutting KEPs).
     - [PreBind](#prebind)
     - [Unreserve](#unreserve)
   - [kubelet](#kubelet)
-    - [Managing resources](#managing-resources)
     - [Communication between kubelet and resource kubelet plugin](#communication-between-kubelet-and-resource-kubelet-plugin)
-      - [NodeListAndWatchResources](#nodelistandwatchresources)
+    - [Version skew](#version-skew)
+    - [Security](#security)
+    - [Managing resources](#managing-resources)
       - [NodePrepareResource](#nodeprepareresource)
       - [NodeUnprepareResources](#nodeunprepareresources)
   - [Simulation with CA](#simulation-with-ca)
@@ -526,25 +527,35 @@ In production, a similar PodTemplateSpec in a Deployment will be used.
 ### Publishing node resources
 
 The resources available on a node need to be published to the API server. In
-the typical case, this is expected to be published by the on-node driver via
-the kubelet, as described below. However, the source of this data may vary; for
+the typical case, this is expected to be published by the on-node driver
+as described in the next paragraph. However, the source of this data may vary; for
 example, a cloud provider controller could populate this based upon information
 from the cloud provider API.
 
-In the kubelet case, each kubelet publishes kubelet publishes a set of
-`ResourceSlice` objects to the API server with content provided by the
-corresponding DRA drivers running on its node. Access control through the node
-authorizer ensures that the kubelet running on one node is not allowed to
-create or modify `ResourceSlices` belonging to another node. A `nodeName`
-field in each `ResourceSlice` object is used to determine which objects are
-managed by which kubelet.
-
-**NOTE:**  `ResourceSlices` are published separately for each driver, using
-whatever version of the `resource.k8s.io` API is supported by the kubelet. That
-same version is then also used in the gRPC interface between the kubelet and
-the DRA drivers providing content for those objects. It might be possible to
-support version skew (= keeping kubelet at an older version than the control
-plane and the DRA drivers) in the future, but currently this is out of scope.
+In the kubelet case, each driver running on a node publishes a set of
+`ResourceSlice` objects to the API server for its own resources, using its
+connection to the apiserver. Access control through a validating admission
+policy can ensure that the drivers running on one node are not allowed to
+create or modify `ResourceSlices` belonging to another node. The `nodeName`
+and `driverName` fields in each `ResourceSlice` object are used to determine which objects are
+managed by which driver instance. The owner reference ensures that objects
+beloging to a node get cleaned up when the node gets removed.
+
+In addition, whenever kubelet starts, it first deletes all `ResourceSlices`
+belonging to the node with a `DeleteCollection` call that uses the node name in
+a field filter. This ensures that no pods depending in DRA get scheduled to the
+node until the required DRA drivers have started up again (node reboot) and
+reconnected to kubelet (kubelet restart). It also ensures that drivers which
+don't get started up again at all don't leave stale `ResourceSlices`
+behind. Garbage collection does not help in this case because the node object
+still exists. For the same reasons, the ResourceSlices belonging to a driver
+get removed when the driver unregisters, this time with a field filter for node
+name and driver name.
+
+Deleting `ResourceSlices` is possible because all information in them can be
+reconstructed by the driver. This has no effect on already allocated claims
+because the allocation result is tracked in those claims, not the
+`ResourceSlice` objects (see [below](#state-and-communication)).
 
 Embedded inside each `ResourceSlice` is the representation of the resources
 managed by a driver according to a specific "structured model". In the example
@@ -842,7 +853,7 @@ return quickly without doing any work for pods.
 
 #### Compromised node
 
-Kubelet is intentionally limited to read-only access for ResourceClass and ResourceClaim
+Kubelet is intentionally limited to read-only access for ResourceClaim
 to prevent that a
 compromised kubelet interferes with scheduling of pending pods, for example
 by updating status information normally set by the scheduler.
@@ -863,22 +874,24 @@ allocation, such an attack is still possible, but the attack code would have to
 be different for each resource driver because all of them will use structured
 parameters differently for reporting resource availability.
 
-#### Compromised resource driver plugin
+#### Compromised kubelet plugin
 
 This is the result of an attack against the resource driver, either from a
 container which uses a resource exposed by the driver, a compromised kubelet
 which interacts with the plugin, or due to resource driver running on a node
 with a compromised root account.
 
-The resource driver plugin only needs read access to objects described in this
-KEP, so compromising it does not interfere with dynamic resource allocation for
-other drivers.
+The resource driver needs write access for ResourceSlices. It can be deployed so
+that it can only write objects associated with the node, so the impact of a
+compromise would be limited to the node. Other drivers on the node could also
+be impacted because there is no separation by driver.
 
-A resource driver may need root access on the node to manage
+However, a resource driver may need root access on the node to manage
 hardware. Attacking the driver therefore may lead to root privilege
 escalation. Ideally, driver authors should try to avoid depending on root
 permissions and instead use capabilities or special permissions for the kernel
-APIs that they depend on.
+APIs that they depend on. Long term, limiting apiserver access by driver
+name would be useful.
 
 A resource driver may also need privileged access to remote services to manage
 network-attached devices. Resource driver vendors and cluster administrators
@@ -931,7 +944,7 @@ Several components must be implemented or modified in Kubernetes:
   ResourceClaim (directly or through a template) and ensure that the
   resource is allocated before the Pod gets scheduled, similar to
   https://github.com/kubernetes/kubernetes/blob/master/pkg/controller/volume/scheduling/scheduler_binder.go
-- Kubelet must be extended to retrieve information from ResourceClaims
+- Kubelet must be extended to manage ResourceClaims
   and to call a resource kubelet plugin. That plugin returns CDI device ID(s)
   which then must be passed to the container runtime.
 
@@ -1188,13 +1201,13 @@ drivers are expected to be written for Kubernetes.
 
 ##### ResourceSlice
 
-For each node, one or more ResourceSlice objects get created. The kubelet
-publishes them with the node as the owner, so they get deleted when a node goes
+For each node, one or more ResourceSlice objects get created. The drivers
+on a node publish them with the node as the owner, so they get deleted when a node goes
 down and then gets removed.
 
 All list types are atomic because that makes tracking the owner for
 server-side-apply (SSA) simpler. Patching individual list elements is not
-needed and there is a single owner (kubelet).
+needed and there is a single owner.
 
 ```go
 // ResourceSlice provides information about available
@@ -2049,6 +2062,53 @@ Unreserve is called in two scenarios:
 
 ### kubelet
 
+#### Communication between kubelet and resource kubelet plugin
+
+Resource kubelet plugins are discovered through the [kubelet plugin registration
+mechanism](https://kubernetes.io/docs/concepts/extend-kubernetes/compute-storage-net/device-plugins/#device-plugin-registration). A
+new "ResourcePlugin" type will be used in the Type field of the
+[PluginInfo](https://pkg.go.dev/k8s.io/kubelet/pkg/apis/pluginregistration/v1#PluginInfo)
+response to distinguish the plugin from device and CSI plugins.
+
+Under the advertised Unix Domain socket the kubelet plugin provides the
+k8s.io/kubelet/pkg/apis/dra gRPC interface. It was inspired by
+[CSI](https://github.com/container-storage-interface/spec/blob/master/spec.md),
+with “volume” replaced by “resource” and volume specific parts removed.
+
+#### Version skew
+
+Previously, kubelet retrieved ResourceClaims and published ResourceSlices on
+behalf of DRA drivers on the node. The information included in those got passed
+between API server, kubelet, and kubelet plugin using the version of the
+resource.k8s.io used by the kubelet. Combining a kubelet using some older API
+version with a plugin using a new version was not possible because conversion
+of the resource.k8s.io types is only supported in the API server and an old
+kubelet wouldn't know about a new version anyway.
+
+Keeping kubelet at some old release while upgrading the control and DRA drivers
+is desirable and officially supported by Kubernetes. To support the same when
+using DRA, the kubelet now leaves [ResourceSlice
+handling](#publishing-node-resources) almost entirely to the plugins. The
+remaining calls are done with whatever resource.k8s.io API version is the
+latest known to the kubelet. To support version skew, support for older API
+versions must be preserved as far back as support for older kubelet releases is
+desired.
+
+#### Security
+
+The daemonset of a DRA driver must be configured to have a service account
+which grants the following permissions:
+- get/list/watch/create/update/oatch/delete ResourceSlice
+- get ResourceClaim
+- get Node
+
+Ideally, write access to ResourceSlice should be limited to objects belonging
+to the node. This is possible with a [validating admission
+policy](https://kubernetes.io/docs/reference/access-authn-authz/validating-admission-policy/). As
+this is not a core feature of the DRA KEP, instructions for how to do that will
+not be included here. Instead, the DRA example driver will provide an example
+and documentation.
+
 #### Managing resources
 
 kubelet must ensure that resources are ready for use on the node before running
@@ -2068,53 +2128,18 @@ successfully before allowing the pod to be deleted. This ensures that network-at
 for other Pods, including those that might get scheduled to other nodes. It
 also signals that it is safe to deallocate and delete the ResourceClaim.
 
-
 ![kubelet](./kubelet.png)
 
-#### Communication between kubelet and resource kubelet plugin
-
-Resource kubelet plugins are discovered through the [kubelet plugin registration
-mechanism](https://kubernetes.io/docs/concepts/extend-kubernetes/compute-storage-net/device-plugins/#device-plugin-registration). A
-new "ResourcePlugin" type will be used in the Type field of the
-[PluginInfo](https://pkg.go.dev/k8s.io/kubelet/pkg/apis/pluginregistration/v1#PluginInfo)
-response to distinguish the plugin from device and CSI plugins.
-
-Under the advertised Unix Domain socket the kubelet plugin provides the
-k8s.io/kubelet/pkg/apis/dra gRPC interface. It was inspired by
-[CSI](https://github.com/container-storage-interface/spec/blob/master/spec.md),
-with “volume” replaced by “resource” and volume specific parts removed.
-
-##### NodeListAndWatchResources
-
-NodeListAndWatchResources returns a stream of NodeResourcesResponse objects.
-At the start and whenever resource availability changes, the
-plugin must send one such object with all information to the kubelet. The
-kubelet then syncs that information with ResourceSlice objects.
-
-```
-message NodeListAndWatchResourcesRequest {
-}
-
-message NodeListAndWatchResourcesResponse {
-    repeated k8s.io.api.resource.v1alpha2.ResourceModel resources = 1;
-}
-```
-
 ##### NodePrepareResource
 
 This RPC is called by the kubelet when a Pod that wants to use the specified
 resource is scheduled on a node. The Plugin SHALL assume that this RPC will be
 executed on the node where the resource will be used.
 
-ResourceClaim.meta.Namespace, ResourceClaim.meta.UID, ResourceClaim.Name and
-one of the ResourceHandles from the ResourceClaimStatus.AllocationResult with
-a matching DriverName should be passed to the Plugin as parameters to identify
+ResourceClaim.meta.Namespace, ResourceClaim.meta.UID, ResourceClaim.Name are
+passed to the Plugin as parameters to identify
 the claim and perform resource preparation.
 
-ResourceClaim parameters (namespace, UUID, name) are useful for debugging.
-They enable the Plugin to retrieve the full ResourceClaim object, should it
-ever be needed (normally it shouldn't).
-
 The Plugin SHALL return fully qualified device name[s].
 
 The Plugin SHALL ensure that there are json file[s] in CDI format
@@ -2155,20 +2180,16 @@ message Claim {
     // The name of the Resource claim (ResourceClaim.meta.Name)
     // This field is REQUIRED.
     string name = 3;
-    // Resource handle (AllocationResult.ResourceHandles[*].Data)
-    // This field is OPTIONAL.
-    string resource_handle = 4;
-    // Structured parameter resource handle (AllocationResult.ResourceHandles[*].StructuredData).
-    // This field is OPTIONAL. If present, it needs to be used
-    // instead of resource_handle. It will only have a single entry.
-    //
-    // Using "repeated" instead of "optional" is a workaround for https://github.com/gogo/protobuf/issues/713.
-    repeated k8s.io.api.resource.v1alpha2.StructuredResourceHandle structured_resource_handle = 5;
 }
 ```
 
-`resource_handle` and `structured_resource_handle` will be set depending on how
-the claim was allocated. See also KEP #3063.
+The allocation result is intentionally not included here. The content of that
+field is version-dependent. The kubelet would need to discover in which version
+each plugin wants the data, then potentially get the claim multiple times
+because only the apiserver can convert between versions. Instead, each plugin
+is required to get the claim itself using its own credentials. In the most common
+case of one plugin per claim, that doubles the number of GETs for each claim
+(once by the kubelet, once by the plugin).
 
 ```
 message NodePrepareResourcesResponse {