From 920f205c17c54ae4bc7dc4ae4c9769d312573101 Mon Sep 17 00:00:00 2001 From: Jon Huhn Date: Wed, 6 Aug 2025 15:28:18 -0500 Subject: [PATCH 1/6] Copy KEP template --- .../README.md | 830 ++++++++++++++++++ .../kep.yaml | 51 ++ 2 files changed, 881 insertions(+) create mode 100644 keps/sig-node/5283-dra-resourceslice-status-device-health/README.md create mode 100644 keps/sig-node/5283-dra-resourceslice-status-device-health/kep.yaml diff --git a/keps/sig-node/5283-dra-resourceslice-status-device-health/README.md b/keps/sig-node/5283-dra-resourceslice-status-device-health/README.md new file mode 100644 index 00000000000..e136e0f33e8 --- /dev/null +++ b/keps/sig-node/5283-dra-resourceslice-status-device-health/README.md @@ -0,0 +1,830 @@ + +# KEP-NNNN: Your short, descriptive title + + + + + + +- [Release Signoff Checklist](#release-signoff-checklist) +- [Summary](#summary) +- [Motivation](#motivation) + - [Goals](#goals) + - [Non-Goals](#non-goals) +- [Proposal](#proposal) + - [User Stories (Optional)](#user-stories-optional) + - [Story 1](#story-1) + - [Story 2](#story-2) + - [Notes/Constraints/Caveats (Optional)](#notesconstraintscaveats-optional) + - [Risks and Mitigations](#risks-and-mitigations) +- [Design Details](#design-details) + - [Test Plan](#test-plan) + - [Prerequisite testing updates](#prerequisite-testing-updates) + - [Unit tests](#unit-tests) + - [Integration tests](#integration-tests) + - [e2e tests](#e2e-tests) + - [Graduation Criteria](#graduation-criteria) + - [Upgrade / Downgrade Strategy](#upgrade--downgrade-strategy) + - [Version Skew Strategy](#version-skew-strategy) +- [Production Readiness Review Questionnaire](#production-readiness-review-questionnaire) + - [Feature Enablement and Rollback](#feature-enablement-and-rollback) + - [Rollout, Upgrade and Rollback Planning](#rollout-upgrade-and-rollback-planning) + - [Monitoring Requirements](#monitoring-requirements) + - [Dependencies](#dependencies) + - [Scalability](#scalability) + - [Troubleshooting](#troubleshooting) +- [Implementation History](#implementation-history) +- [Drawbacks](#drawbacks) +- [Alternatives](#alternatives) +- [Infrastructure Needed (Optional)](#infrastructure-needed-optional) + + +## Release Signoff Checklist + + + +Items marked with (R) are required *prior to targeting to a milestone / release*. + +- [ ] (R) Enhancement issue in release milestone, which links to KEP dir in [kubernetes/enhancements] (not the initial KEP PR) +- [ ] (R) KEP approvers have approved the KEP status as `implementable` +- [ ] (R) Design details are appropriately documented +- [ ] (R) Test plan is in place, giving consideration to SIG Architecture and SIG Testing input (including test refactors) + - [ ] e2e Tests for all Beta API Operations (endpoints) + - [ ] (R) Ensure GA e2e tests meet requirements for [Conformance Tests](https://github.com/kubernetes/community/blob/master/contributors/devel/sig-architecture/conformance-tests.md) + - [ ] (R) Minimum Two Week Window for GA e2e tests to prove flake free +- [ ] (R) Graduation criteria is in place + - [ ] (R) [all GA Endpoints](https://github.com/kubernetes/community/pull/1806) must be hit by [Conformance Tests](https://github.com/kubernetes/community/blob/master/contributors/devel/sig-architecture/conformance-tests.md) +- [ ] (R) Production readiness review completed +- [ ] (R) Production readiness review approved +- [ ] "Implementation History" section is up-to-date for milestone +- [ ] User-facing documentation has been created in [kubernetes/website], for publication to [kubernetes.io] +- [ ] Supporting documentation—e.g., additional design documents, links to mailing list discussions/SIG meetings, relevant PRs/issues, release notes + + + +[kubernetes.io]: https://kubernetes.io/ +[kubernetes/enhancements]: https://git.k8s.io/enhancements +[kubernetes/kubernetes]: https://git.k8s.io/kubernetes +[kubernetes/website]: https://git.k8s.io/website + +## Summary + + + +## Motivation + + + +### Goals + + + +### Non-Goals + + + +## Proposal + + + +### User Stories (Optional) + + + +#### Story 1 + +#### Story 2 + +### Notes/Constraints/Caveats (Optional) + + + +### Risks and Mitigations + + + +## Design Details + + + +### Test Plan + + + +[ ] I/we understand the owners of the involved components may require updates to +existing tests to make this code solid enough prior to committing the changes necessary +to implement this enhancement. + +##### Prerequisite testing updates + + + +##### Unit tests + + + + + +- ``: `` - `` + +##### Integration tests + + + + + +- [test name](https://github.com/kubernetes/kubernetes/blob/2334b8469e1983c525c0c6382125710093a25883/test/integration/...): [integration master](https://testgrid.k8s.io/sig-release-master-blocking#integration-master?include-filter-by-regex=MyCoolFeature), [triage search](https://storage.googleapis.com/k8s-triage/index.html?test=MyCoolFeature) + +##### e2e tests + + + +- [test name](https://github.com/kubernetes/kubernetes/blob/2334b8469e1983c525c0c6382125710093a25883/test/e2e/...): [SIG ...](https://testgrid.k8s.io/sig-...?include-filter-by-regex=MyCoolFeature), [triage search](https://storage.googleapis.com/k8s-triage/index.html?test=MyCoolFeature) + +### Graduation Criteria + + + +### Upgrade / Downgrade Strategy + + + +### Version Skew Strategy + + + +## Production Readiness Review Questionnaire + + + +### Feature Enablement and Rollback + + + +###### How can this feature be enabled / disabled in a live cluster? + + + +- [ ] Feature gate (also fill in values in `kep.yaml`) + - Feature gate name: + - Components depending on the feature gate: +- [ ] Other + - Describe the mechanism: + - Will enabling / disabling the feature require downtime of the control + plane? + - Will enabling / disabling the feature require downtime or reprovisioning + of a node? + +###### Does enabling the feature change any default behavior? + + + +###### Can the feature be disabled once it has been enabled (i.e. can we roll back the enablement)? + + + +###### What happens if we reenable the feature if it was previously rolled back? + +###### Are there any tests for feature enablement/disablement? + + + +### Rollout, Upgrade and Rollback Planning + + + +###### How can a rollout or rollback fail? Can it impact already running workloads? + + + +###### What specific metrics should inform a rollback? + + + +###### Were upgrade and rollback tested? Was the upgrade->downgrade->upgrade path tested? + + + +###### Is the rollout accompanied by any deprecations and/or removals of features, APIs, fields of API types, flags, etc.? + + + +### Monitoring Requirements + + + +###### How can an operator determine if the feature is in use by workloads? + + + +###### How can someone using this feature know that it is working for their instance? + + + +- [ ] Events + - Event Reason: +- [ ] API .status + - Condition name: + - Other field: +- [ ] Other (treat as last resort) + - Details: + +###### What are the reasonable SLOs (Service Level Objectives) for the enhancement? + + + +###### What are the SLIs (Service Level Indicators) an operator can use to determine the health of the service? + + + +- [ ] Metrics + - Metric name: + - [Optional] Aggregation method: + - Components exposing the metric: +- [ ] Other (treat as last resort) + - Details: + +###### Are there any missing metrics that would be useful to have to improve observability of this feature? + + + +### Dependencies + + + +###### Does this feature depend on any specific services running in the cluster? + + + +### Scalability + + + +###### Will enabling / using this feature result in any new API calls? + + + +###### Will enabling / using this feature result in introducing new API types? + + + +###### Will enabling / using this feature result in any new calls to the cloud provider? + + + +###### Will enabling / using this feature result in increasing size or count of the existing API objects? + + + +###### Will enabling / using this feature result in increasing time taken by any operations covered by existing SLIs/SLOs? + + + +###### Will enabling / using this feature result in non-negligible increase of resource usage (CPU, RAM, disk, IO, ...) in any components? + + + +###### Can enabling / using this feature result in resource exhaustion of some node resources (PIDs, sockets, inodes, etc.)? + + + +### Troubleshooting + + + +###### How does this feature react if the API server and/or etcd is unavailable? + +###### What are other known failure modes? + + + +###### What steps should be taken if SLOs are not being met to determine the problem? + +## Implementation History + + + +## Drawbacks + + + +## Alternatives + + + +## Infrastructure Needed (Optional) + + diff --git a/keps/sig-node/5283-dra-resourceslice-status-device-health/kep.yaml b/keps/sig-node/5283-dra-resourceslice-status-device-health/kep.yaml new file mode 100644 index 00000000000..5dfddc15e73 --- /dev/null +++ b/keps/sig-node/5283-dra-resourceslice-status-device-health/kep.yaml @@ -0,0 +1,51 @@ +title: KEP Template +kep-number: NNNN +authors: + - "@jane.doe" +owning-sig: sig-xyz +participating-sigs: + - sig-aaa + - sig-bbb +status: provisional|implementable|implemented|deferred|rejected|withdrawn|replaced +creation-date: yyyy-mm-dd +reviewers: + - TBD + - "@alice.doe" +approvers: + - TBD + - "@oscar.doe" + +see-also: + - "/keps/sig-aaa/1234-we-heard-you-like-keps" + - "/keps/sig-bbb/2345-everyone-gets-a-kep" +replaces: + - "/keps/sig-ccc/3456-replaced-kep" + +# The target maturity stage in the current dev cycle for this KEP. +# If the purpose of this KEP is to deprecate a user-visible feature +# and a Deprecated feature gates are added, they should be deprecated|disabled|removed. +stage: alpha|beta|stable + +# The most recent milestone for which work toward delivery of this KEP has been +# done. This can be the current (upcoming) milestone, if it is being actively +# worked on. +latest-milestone: "v1.19" + +# The milestone at which this feature was, or is targeted to be, at each stage. +milestone: + alpha: "v1.19" + beta: "v1.20" + stable: "v1.22" + +# The following PRR answers are required at alpha release +# List the feature gate name and the components for which it must be enabled +feature-gates: + - name: MyFeature + components: + - kube-apiserver + - kube-controller-manager +disable-supported: true + +# The following PRR answers are required at beta release +metrics: + - my_feature_metric From 44c3a803b40f4b6c48d8ed786acc59213554e3e8 Mon Sep 17 00:00:00 2001 From: Jon Huhn Date: Wed, 6 Aug 2025 15:27:13 -0500 Subject: [PATCH 2/6] amend! 920f205c17c54ae4bc7dc4ae4c9769d312573101 KEP-5283: DRA: ResourceSlice Status for Device Health Tracking --- .../README.md | 223 +++++++++++++++++- .../kep.yaml | 38 +-- 2 files changed, 225 insertions(+), 36 deletions(-) diff --git a/keps/sig-node/5283-dra-resourceslice-status-device-health/README.md b/keps/sig-node/5283-dra-resourceslice-status-device-health/README.md index e136e0f33e8..092a11f0d69 100644 --- a/keps/sig-node/5283-dra-resourceslice-status-device-health/README.md +++ b/keps/sig-node/5283-dra-resourceslice-status-device-health/README.md @@ -3,19 +3,19 @@ To get started with this template: -- [ ] **Pick a hosting SIG.** +- [X] **Pick a hosting SIG.** Make sure that the problem space is something the SIG is interested in taking up. KEPs should not be checked in without a sponsoring SIG. -- [ ] **Create an issue in kubernetes/enhancements** +- [X] **Create an issue in kubernetes/enhancements** When filing an enhancement tracking issue, please make sure to complete all fields in that template. One of the fields asks for a link to the KEP. You can leave that blank until this KEP is filed, and then go back to the enhancement and add the link. -- [ ] **Make a copy of this template directory.** +- [X] **Make a copy of this template directory.** Copy this template into the owning SIG's directory and name it `NNNN-short-descriptive-title`, where `NNNN` is the issue number (with no leading-zero padding) assigned to your enhancement above. -- [ ] **Fill out as much of the kep.yaml file as you can.** +- [X] **Fill out as much of the kep.yaml file as you can.** At minimum, you should fill in the "Title", "Authors", "Owning-sig", "Status", and date-related fields. - [ ] **Fill out this file as best you can.** @@ -58,7 +58,7 @@ If none of those approvers are still appropriate, then changes to that list should be approved by the remaining approvers and/or the owning SIG (or SIG Architecture for cross-cutting KEPs). --> -# KEP-NNNN: Your short, descriptive title +# KEP-5283: DRA: ResourceSlice Status for Device Health Tracking @@ -173,6 +176,14 @@ updates. [documentation style guide]: https://github.com/kubernetes/community/blob/master/contributors/guide/style-guide.md --> +With Dynamic Resource Allocation (DRA), devices that can be allocated to Pods +are advertised in ResourceSlices. Device entries in a ResourceSlice describe +properties like identifiers, capacity, and firmware or kernel driver versions. + +Sometimes devices fail and an automated agent or cluster administrator must take +some action to restore the device to a healthy state. This KEP defines a +standard way to determine whether or not a device is considered healthy. + ## Motivation +Since the DRA APIs currently do not expose a standard way to classify device +health, cluster administrators are tasked with finding other means of doing that +for each kind of device that is defined in a ResourceSlice in order to identify +and restore unhealthy devices back to their full functionality. + +When unhealthy devices are no longer able to support new or existing workloads, +DRA drivers and cluster administrators already have the option to +[taint](https://kep.k8s.io/5055) the devices. A standard representation of +device health in the ResourceSlice API is needed to express the state of the +devices in cases where the side effects (`NoSchedule`/`NoExecute`) of taints are +not desired. + ### Goals +- Expose well-formed data in the ResourceSlice API representing the real-time + health of the devices a ResourceSlice describes. + ### Non-Goals +- Define what constitutes a "healthy" or "unhealthy" device. That distinction is + made by each DRA driver. +- Automatically take action based on the health status of a device. This KEP + only defines how that status is represented in the API. + ## Proposal -### User Stories (Optional) +### User Stories -#### Story 1 +#### Visibility + +As a cluster administrator, I want to determine the overall health of the +DRA-exposed devices available throughout my cluster. -#### Story 2 +#### Enabling Automated Remediation + +As a cluster administrator, I want to determine how to remediate unhealthy +devices when different failure modes require different methods of remediation. ### Notes/Constraints/Caveats (Optional) @@ -254,6 +291,64 @@ required) or even code snippets. If there's any ambiguity about HOW your proposal will be implemented, this is the place to discuss them. --> +A new `status` field will be added to the ResourceSlice API, along with fields +for health information for each device listed in the ResourceSlice: + +```go +type ResourceSlice struct { + ... + + // Contains the status observed by the driver. + // +optional + Status ResourceSliceStatus `json:"status,omitempty"` +} + +// ResourceSliceStatus contains the observed status of a ResourceSlice. +type ResourceSliceStatus struct { + // Devices contains the status of each device. + // +optional + Devices []DeviceStatus `json:"devices"` +} + +// DeviceStatus represents the status of a [Device]. +type DeviceStatus struct { + // Name maps this status to the device listed in .spec.devices. + // + // +required + Name string `json:"name"` + + // Health contains the health of a device as observed by the driver. + // + // +optional + Health *DeviceHealthStatus `json:"health,omitempty"` +} + +// DeviceHealthState represents the overall health status of a device. +type DeviceHealthState string + +// These are valid conditions of a device. +const ( + // DeviceHealthy is the state for a healthy device. + DeviceHealthy DeviceHealthState = "Healthy" + // DeviceUnhealthy is the state for a unhealthy device. + DeviceUnhealthy DeviceHealthState = "Unhealthy" + // DeviceUnknown is the state for a device whose health is unknown. + DeviceUnknown DeviceHealthState = "Unknown" +) + +// DeviceHealthStatus represents the health of a device as observed by the driver. +type DeviceHealthStatus struct { + // State is the overall health status of a device. + // + // +required + State DeviceHealthState `json:"state"` +} +``` + +A dedicated place for health information for devices is flexible enough to +accommodate new fields related to health, like describing particular failure +modes or remediation methods. + ### Test Plan +### Device Conditions + +Instead of inventing new API structures, [`metav1.Condition`]s could represent +the health status of each device. The `DeviceStatus` type could then look like +the following: + +```go +// DeviceStatus represents the status of a [Device]. +type DeviceStatus struct { + // Name maps this status to the device listed in .spec.devices. + // + // +required + Name string `json:"name"` + + // Represents the observations of a device's current state. + // Known .status.devices[].conditions.type are: "Available" + // +patchMergeKey=type + // +patchStrategy=merge + // +listType=map + // +listMapKey=type + Conditions []metav1.Condition `json:"conditions,omitempty" patchStrategy:"merge" patchMergeKey:"type"` +} + +// These are valid conditions of a device. +const ( + // Available means the device is available, as determined by the driver. + DeviceAvailable = "Available" +) +``` + +The `Available` condition would be the recommended way to declare the overall +health of a device. Other condition types like `Degraded` could also be added. + +While users are likely already familiar with the concept of these conditions, +having multiple lists of conditions in one ResourceSlice is not a pattern that +exists elsewhere in Kubernetes. Existing tools and libraries used to looking at +`status.conditions` may not work when there are separate instances in each of +`status.devices[].conditions`. + +The fields of each individual condition may also not satisfy the use cases +targeted by this KEP. The single `reason` and `message` fields may not be able +to capture all of the extra context behind a particular health state of a +device. The `observedGeneration` field may also not be useful since factors +outside the ResourceSlice may play into the health of a device more than +parameters in the ResourceSlice's own `spec`, so a condition's +`observedGeneration` matching the ResourceSlice's `metadata.generation` may not +reliably indicate that the device is healthy at that exact moment. + +### Standardized Attributes + +A set of standard attributes could be defined to represent device health. + +e.g. +```yaml +kind: ResourceSlice +apiVersion: resource.k8s.io/v1 +... +spec: + driver: cards.dra.example.com + devices: + - name: card-1 + attributes: + manufacturer: # a vendor-specific attribute, automatically qualified as cards.dra.example.com/manufacturer + string: Cards-are-us Inc. + resource.kubernetes.io/health: # a standard attribute expressing overall health status + string: Unhealthy +``` + +This approach requires no new API structure and DRA drivers already assume +ownership over device attributes. Users can also utilize attributes in +ResourceClaims to decide whether or not unhealthy devices are acceptable for a +request if a driver doesn't express its own opinion by also tainting the device. + +However, the lack of a dedicated place in the API for this information makes it +less discoverable. Users will be less likely to look for health status in a set +of arbitrary `attributes` than a list of [`metav1.Condition`]s or a field +labeled `health`. Other related information like the particular failure mode or +remediation strategy of an unhealthy device also becomes mixed in with all of +the other attributes of a device, making their relationship less clear even if +the attributes have similar names. + +### Standardized Events + +DRA drivers could be expected to generate Events to indicate health status for +devices. One such Event might look like the following: + +``` +LAST SEEN TYPE REASON OBJECT MESSAGE +2m4s Warning DeviceUnhealthy resourceslice/node-1 Device 'card-1' is Unhealthy. +``` + +Like standard attributes, this approach requires no new API structure, but is +similarly less discoverable and doesn't enable bundling related information in a +structured way. Health information from Events cannot be used to make scheduling +decisions in ResourceClaims. One benefit of using Events over other approaches +is that existing alerting systems may already be configured to surface when +devices become unhealthy based on the Event's `type`. + +Events are also ephemeral, so it isn't possible to determine if a device is +healthy at the present moment since a `DeviceUnhealthy` Event may have occurred +a while ago and since expired. The Event API is described as "informative, +best-effort, supplemental data" which is likely insufficient by itself for most +use cases for tracking device health. + ## Infrastructure Needed (Optional) + +[`metav1.Condition`]: https://pkg.go.dev/k8s.io/apimachinery/pkg/apis/meta/v1#Condition diff --git a/keps/sig-node/5283-dra-resourceslice-status-device-health/kep.yaml b/keps/sig-node/5283-dra-resourceslice-status-device-health/kep.yaml index 5dfddc15e73..51c24b512b5 100644 --- a/keps/sig-node/5283-dra-resourceslice-status-device-health/kep.yaml +++ b/keps/sig-node/5283-dra-resourceslice-status-device-health/kep.yaml @@ -1,51 +1,39 @@ -title: KEP Template -kep-number: NNNN +title: "DRA: ResourceSlice Status for Device Health Tracking" +kep-number: 5283 authors: - - "@jane.doe" -owning-sig: sig-xyz + - "@nojnhuh" +owning-sig: sig-node participating-sigs: - - sig-aaa - - sig-bbb -status: provisional|implementable|implemented|deferred|rejected|withdrawn|replaced -creation-date: yyyy-mm-dd + - sig-scheduling +status: provisional +creation-date: 2025-08-06 reviewers: - - TBD - - "@alice.doe" + - "@johnbelamaric" approvers: - TBD - - "@oscar.doe" see-also: - - "/keps/sig-aaa/1234-we-heard-you-like-keps" - - "/keps/sig-bbb/2345-everyone-gets-a-kep" -replaces: - - "/keps/sig-ccc/3456-replaced-kep" + - "/keps/sig-scheduling/5055-dra-device-taints-and-tolerations/kep.yaml" # The target maturity stage in the current dev cycle for this KEP. -# If the purpose of this KEP is to deprecate a user-visible feature -# and a Deprecated feature gates are added, they should be deprecated|disabled|removed. -stage: alpha|beta|stable +stage: alpha # The most recent milestone for which work toward delivery of this KEP has been # done. This can be the current (upcoming) milestone, if it is being actively # worked on. -latest-milestone: "v1.19" +latest-milestone: "v1.35" # The milestone at which this feature was, or is targeted to be, at each stage. milestone: - alpha: "v1.19" - beta: "v1.20" - stable: "v1.22" + alpha: "v1.35" # The following PRR answers are required at alpha release # List the feature gate name and the components for which it must be enabled feature-gates: - - name: MyFeature + - name: DRAResourceSliceStatusDeviceHealth components: - kube-apiserver - - kube-controller-manager disable-supported: true # The following PRR answers are required at beta release metrics: - - my_feature_metric From bebbb870e2af1f7d7552cd407e0d1f8fc6a02274 Mon Sep 17 00:00:00 2001 From: Jon Huhn Date: Tue, 12 Aug 2025 10:39:48 -0500 Subject: [PATCH 3/6] fixup! Copy KEP template Add TODO to fill out API --- .../README.md | 20 +++---------------- 1 file changed, 3 insertions(+), 17 deletions(-) diff --git a/keps/sig-node/5283-dra-resourceslice-status-device-health/README.md b/keps/sig-node/5283-dra-resourceslice-status-device-health/README.md index 092a11f0d69..6d5bdb12f84 100644 --- a/keps/sig-node/5283-dra-resourceslice-status-device-health/README.md +++ b/keps/sig-node/5283-dra-resourceslice-status-device-health/README.md @@ -323,25 +323,11 @@ type DeviceStatus struct { Health *DeviceHealthStatus `json:"health,omitempty"` } -// DeviceHealthState represents the overall health status of a device. -type DeviceHealthState string - -// These are valid conditions of a device. -const ( - // DeviceHealthy is the state for a healthy device. - DeviceHealthy DeviceHealthState = "Healthy" - // DeviceUnhealthy is the state for a unhealthy device. - DeviceUnhealthy DeviceHealthState = "Unhealthy" - // DeviceUnknown is the state for a device whose health is unknown. - DeviceUnknown DeviceHealthState = "Unknown" -) - // DeviceHealthStatus represents the health of a device as observed by the driver. type DeviceHealthStatus struct { - // State is the overall health status of a device. - // - // +required - State DeviceHealthState `json:"state"` + // TODO + // This could contain fields representing the overall "Healthy" or "Unhealthy" + // status and other context around the reason for that status. } ``` From 9a961e537c4f01097a84e2610c83a51d88548b30 Mon Sep 17 00:00:00 2001 From: Jon Huhn Date: Wed, 13 Aug 2025 15:19:22 -0500 Subject: [PATCH 4/6] fixup! Copy KEP template Add alternative for vendor-provided metrics --- .../README.md | 24 +++++++++++++++++++ 1 file changed, 24 insertions(+) diff --git a/keps/sig-node/5283-dra-resourceslice-status-device-health/README.md b/keps/sig-node/5283-dra-resourceslice-status-device-health/README.md index 6d5bdb12f84..d6cefb0263c 100644 --- a/keps/sig-node/5283-dra-resourceslice-status-device-health/README.md +++ b/keps/sig-node/5283-dra-resourceslice-status-device-health/README.md @@ -110,6 +110,7 @@ tags, and then generate with `hack/update-toc.sh`. - [Device Conditions](#device-conditions) - [Standardized Attributes](#standardized-attributes) - [Standardized Events](#standardized-events) + - [Vendor-Provided Metrics](#vendor-provided-metrics) - [Infrastructure Needed (Optional)](#infrastructure-needed-optional) @@ -1006,6 +1007,29 @@ a while ago and since expired. The Event API is described as "informative, best-effort, supplemental data" which is likely insufficient by itself for most use cases for tracking device health. +### Vendor-Provided Metrics + +Device vendors could be expected to track and publish metrics related to the +devices they manage and document specific queries that indicate the various +possible health states of the devices. Cluster administrators could then wire +those metrics into existing alerting mechanisms. + +This option provides much more flexibility to vendors in how they express the +health of a device than the more rigid APIs defined by the other options. +Vendor-defined metrics can represent a wider variety of dimensions and degrees +of health at a higher resolution than would be possible with Kubernetes APIs. +Cluster administrators are also afforded more flexibility by the freedom to +interpret metrics in a way that works for their own environments instead of +having to inherit a more binary "Healthy" or "Unhealthy" status from a driver +which may or may not be customizable. + +The main cost of that flexibility is the lack of standardization, where cluster +administrators have to track down from each vendor how to determine if a given +device is in a healthy state as opposed to inspecting a well-defined area of a +vendor-agnostic API like ResourceSlice. This lack of standardization also makes +integrations like generic controllers that automatically taint unhealthy devices +less straightforward to implement. + ## Infrastructure Needed (Optional)