DRA: Provide helper for implementing webhooks for opaque config

[nojnhuh]

What would you like to be added?

A new webhook package in the k8s.io/dynamic-resource-allocation module should implement an abstraction on top of a generic validating webhook that drivers can use to plug in their own logic to validate the opaque config present in ResourceClaims and ResourceClaimTemplates.

Why is this needed?

DRA drivers are currently on their own to implement validating webhooks "from scratch" which involves a non-trivial amount of boilerplate that is equivalent for most drivers. Starting an HTTPS server, reading the AdmissionReview object, extracting the opaque config from the ResourceClaim or ResourceClaimTemplate within, and writing the response can all be shared be shared by the vast majority of DRA drivers, leaving only the validation logic to be defined by each driver.

To illustrate, this snippet dra-example-driver's webhook shows the extent of the driver-specific logic amongst the rest of the webhook implementation: https://github.com/kubernetes-sigs/dra-example-driver/blob/ba82bf55dad820297b124b64cb4487958bb17466/cmd/dra-example-webhook/main.go#L275-L283

The NVIDIA DRA driver has a similar amount of nearly identical boilerplate around its specific logic: https://github.com/NVIDIA/k8s-dra-driver-gpu/blob/7f591c28f3853f432d11985ecc533ed63e472b9f/cmd/webhook/main.go#L259-L284

The dra-example-driver and NVIDIA driver have already gone back and forth a few times sharing improvements to the boilerplate to handle things like natively validating different API versions of ResourceClaims. Being able to more easily collaborate that way on a shared implementation would benefit all drivers looking to add webhooks.

With webhooks also being a key reason why KEP-5322 was deferred beyond v1.35, making it easier for drivers to implement webhooks will help improve DRA driver quality and cluster stability overall.

[k8s-ci-robot]

This issue is currently awaiting triage.

If a SIG or subproject determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[nojnhuh]

/wg device-management
/assign

[nojnhuh]

@kannon92 mentioned that ValidatingAdmissionPolicies are preferable to webhooks. I tried playing around with how that might look for a DRA driver and hacked this together (Kubernetes 1.33+) for the dra-example-driver:

vap.yaml

apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionPolicy
metadata:
  name: dra-example-driver-opaque
spec:
  failurePolicy: Fail
  matchConstraints:
    resourceRules:
    - apiGroups:   ["resource.k8s.io"]
      apiVersions: ["v1beta1", "v1beta2", "v1"]
      operations:  ["CREATE", "UPDATE"]
      resources:   ["resourceclaims", "resourceclaimtemplates"]
  variables:
    - name: spec
      expression: |-
        has(object.spec) ? (has(object.spec.spec) ? object.spec.spec : object.spec) : {}
    - name: specPath
      expression: |-
        has(object.spec) ? (has(object.spec.spec) ? "spec.spec" : "spec") : ""
    - name: configs
      expression: |-
        has(variables.spec.devices) && has(variables.spec.devices.config) ? variables.spec.devices.config : []
    - name: driverConfigs
      expression: |-
        variables.configs.map(
          config,
          config.opaque.driver == "gpu.example.com" ? config : null
        )
    - name: unknownAPIVersions
      expression: |-
        variables.driverConfigs.map(
          config,
          config != null && config.opaque.parameters.apiVersion != "gpu.resource.example.com/v1alpha1" ? config.opaque.parameters.apiVersion : ""
        )
  validations:
    - expression: |-
        variables.unknownAPIVersions.all(invalid, invalid == "")
      messageExpression: |-
        "invalid " + variables.unknownAPIVersions.transformList(i, apiVersion, apiVersion != "", "%s.devices.config[%d].opaque.parameters.apiVersion: '%s'".format([variables.specPath, i, apiVersion])).join(', ')
---
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionPolicyBinding
metadata:
  name: dra-example-driver-opaque
spec:
  policyName: dra-example-driver-opaque
  validationActions: [Deny]

resourceclaimtemplate.yaml

apiVersion: resource.k8s.io/v1
kind: ResourceClaimTemplate
metadata:
  name: multiple-gpus
spec:
  spec:
    devices:
      requests:
      - name: ts-gpu
        exactly:
          deviceClassName: gpu.example.com
      - name: sp-gpu
        exactly:
          deviceClassName: gpu.example.com
      config:
      - requests: ["ts-gpu"]
        opaque:
          driver: gpu.example.com
          parameters:
            apiVersion: gpu.resource.example.com/v1alpha1
            kind: Invalid
            sharing:
              strategy: TimeSlicing
              timeSlicingConfig:
                interval: Long
      - requests: ["sp-gpu"]
        opaque:
          driver: gpu.example.com
          parameters:
            apiVersion: gpu.resource.example.com/v1invalid
            kind: GpuConfig
            sharing:
              strategy: SpacePartitioning
              spacePartitioningConfig:
                partitionCount: 10

The resourceclaimtemplates "multiple-gpus" is invalid: : ValidatingAdmissionPolicy 'dra-example-driver-opaque' with binding 'dra-example-driver-opaque' denied request: invalid spec.spec.devices.config[1].opaque.parameters.apiVersion: 'gpu.resource.example.com/v1invalid'

I think that's enough to convince me that that's a serviceable option. I think it could be a dealbreaker though if drivers need to do the same validation at runtime anyway during NodePrepareResources. That could be easily shared in a driver between a kubelet plugin and a webhook, but not between a kubelet plugin and a ValidatingAdmissionPolicy.

@klueska Could you see ever dropping the validation from NodePrepareResources if that same validation was done up front by a ValidatingAdmissionPolicy? e.g. https://github.com/NVIDIA/k8s-dra-driver-gpu/blob/7f591c28f3853f432d11985ecc533ed63e472b9f/cmd/gpu-kubelet-plugin/device_state.go#L345

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle stale
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale