Merge pull request kubernetes#2431 from logicalhan/metric-keps

expand on GA requirements for metrics stability framework

Author: k8s-ci-robot

keps/prod-readiness/sig-instrumentation/1209.yaml

@@ -1,3 +1,3 @@
 kep-number: 1209
-beta:
+stable:
   approver: "@johnbelamaric"

keps/sig-instrumentation/1209-metrics-stability/README.md

@@ -13,6 +13,19 @@
   - [Upgrade / Downgrade Strategy](#upgrade--downgrade-strategy)
   - [Version Skew Strategy](#version-skew-strategy)
 - [Production Readiness Review Questionnaire](#production-readiness-review-questionnaire)
+    - [How can this feature be enabled / disabled in a live cluster?](#how-can-this-feature-be-enabled--disabled-in-a-live-cluster)
+    - [What specific metrics should inform a rollback?](#what-specific-metrics-should-inform-a-rollback)
+  - [Monitoring Requirements](#monitoring-requirements)
+    - [How can an operator determine if the feature is in use by workloads?](#how-can-an-operator-determine-if-the-feature-is-in-use-by-workloads)
+    - [What are the SLIs (Service Level Indicators) an operator can use to determine the health of the service?](#what-are-the-slis-service-level-indicators-an-operator-can-use-to-determine-the-health-of-the-service)
+    - [Metrics](#metrics)
+  - [Dependencies](#dependencies)
+    - [Does this feature depend on any specific services running in the cluster?](#does-this-feature-depend-on-any-specific-services-running-in-the-cluster)
+    - [For GA, this section is required: approvers should be able to confirm the previous answers based on experience in the field.](#for-ga-this-section-is-required-approvers-should-be-able-to-confirm-the-previous-answers-based-on-experience-in-the-field)
+    - [Will enabling / using this feature result in any new API calls? Describe them, providing:](#will-enabling--using-this-feature-result-in-any-new-api-calls-describe-them-providing)
+  - [Troubleshooting](#troubleshooting)
+    - [How does this feature react if the API server and/or etcd is unavailable?](#how-does-this-feature-react-if-the-api-server-andor-etcd-is-unavailable)
+    - [What are other known failure modes?](#what-are-other-known-failure-modes)
 - [Implementation History](#implementation-history)
 <!-- /toc -->
 
@@ -39,8 +52,8 @@ Items marked with (R) are required *prior to targeting to a milestone / release*
 - [X] (R) Design details are appropriately documented
 - [X] (R) Test plan is in place, giving consideration to SIG Architecture and SIG Testing input
 - [X] (R) Graduation criteria is in place
-- \[Predates\] (R) Production readiness review completed
-- \[Predates\] (R) Production readiness review approved
+- [ ] (R) Production readiness review completed
+- [ ] (R) Production readiness review approved
 - [X] "Implementation History" section is up-to-date for milestone
 - [X] User-facing documentation has been created in [kubernetes/website], for publication to [kubernetes.io]
 - [X] Supporting documentation—e.g., additional design documents, links to mailing list discussions/SIG meetings, relevant PRs/issues, release notes
@@ -56,17 +69,17 @@ This proposal covers the implementation of metrics stability in the kubernetes/k
 
 Historically, the implementation was split into four documents:
 
-1. [Metrics Stability Framework]
-1. [Metrics Stability Migration]
-1. [Metrics Validation and Verification]
-1. [Metrics Stability to Beta]
+1. [Metrics Stability Framework](https://github.com/kubernetes/enhancements/blob/77a84d2d55b5802a615f3fe98e7e7c9bd26c9efc/keps/sig-instrumentation/1209-metrics-stability/20190404-kubernetes-control-plane-metrics-stability.md)
+1. [Metrics Stability Migration](https://github.com/kubernetes/enhancements/blob/77a84d2d55b5802a615f3fe98e7e7c9bd26c9efc/keps/sig-instrumentation/1209-metrics-stability/20190605-metrics-stability-migration.md)
+1. [Metrics Validation and Verification](https://github.com/kubernetes/enhancements/blob/77a84d2d55b5802a615f3fe98e7e7c9bd26c9efc/keps/sig-instrumentation/1209-metrics-stability/20190605-metrics-validation-and-verification.md)
+1. [Metrics Stability to Beta](https://github.com/kubernetes/enhancements/blob/77a84d2d55b5802a615f3fe98e7e7c9bd26c9efc/keps/sig-instrumentation/1209-metrics-stability/20191028-metrics-stability-to-beta.md)
 
 This document is not net new and ties the four together in order to document the lifecycle of this feature.
 
-[Metrics Stability Framework]: keps/sig-instrumentation/1209-metrics-stability/20190404-kubernetes-control-plane-metrics-stability.md
-[Metrics Stability Migration]: keps/sig-instrumentation/1209-metrics-stability/20190605-metrics-stability-migration.md
-[Metrics Validation and Verification]: keps/sig-instrumentation/1209-metrics-stability/20190605-metrics-validation-and-verification.md
-[Metrics Stability to Beta]: keps/sig-instrumentation/1209-metrics-stability/20191028-metrics-stability-to-beta.md
+[Metrics Stability Framework]: 20190404-kubernetes-control-plane-metrics-stability.md
+[Metrics Stability Migration]: 20190605-metrics-stability-migration.md
+[Metrics Validation and Verification]: 20190605-metrics-validation-and-verification.md
+[Metrics Stability to Beta]: 20191028-metrics-stability-to-beta.md
 
 ## Motivation
 
@@ -77,10 +90,10 @@ See:
 1. [Metrics Validation and Verification#Motivation]
 1. [Metrics Stability to Beta#Motivation]
 
-[Metrics Stability Framework#Motivation]: keps/sig-instrumentation/1209-metrics-stability/20190404-kubernetes-control-plane-metrics-stability.md#motivation
-[Metrics Stability Migration#Motivation]: keps/sig-instrumentation/1209-metrics-stability/20190605-metrics-stability-migration.md#motivation
-[Metrics Validation and Verification#Motivation]: keps/sig-instrumentation/1209-metrics-stability/20190605-metrics-validation-and-verification.md#motivation
-[Metrics Stability to Beta#Motivation]: keps/sig-instrumentation/1209-metrics-stability/20191028-metrics-stability-to-beta.md#motivation
+[Metrics Stability Framework#Motivation]: 20190404-kubernetes-control-plane-metrics-stability.md#motivation
+[Metrics Stability Migration#Motivation]: 20190605-metrics-stability-migration.md#motivation
+[Metrics Validation and Verification#Motivation]: 20190605-metrics-validation-and-verification.md#motivation
+[Metrics Stability to Beta#Motivation]: 20191028-metrics-stability-to-beta.md#motivation
 
 ## Proposal
 
@@ -91,10 +104,12 @@ See:
 1. [Metrics Validation and Verification#Proposal]
 1. [Metrics Stability to Beta#Proposal]
 
-[Metrics Stability Framework#Proposal]: keps/sig-instrumentation/1209-metrics-stability/20190404-kubernetes-control-plane-metrics-stability.md#proposal
-[Metrics Stability Migration#General Migration Strategy]: keps/sig-instrumentation/1209-metrics-stability/20190605-metrics-stability-migration.md#general-migration-strategy
-[Metrics Validation and Verification#Proposal]: keps/sig-instrumentation/1209-metrics-stability/20190605-metrics-validation-and-verification.md#proposal
-[Metrics Stability to Beta#Proposal]: keps/sig-instrumentation/1209-metrics-stability/20191028-metrics-stability-to-beta.md#proposal
+https://github.com/kubernetes/enhancements/blob/77a84d2d55b5802a615f3fe98e7e7c9bd26c9efc/keps/sig-instrumentation/1209-metrics-stability/keps/sig-instrumentation/1209-metrics-stability/20190404-kubernetes-control-plane-metrics-stability.md#implementation-history
+
+[Metrics Stability Framework#Proposal]: 20190404-kubernetes-control-plane-metrics-stability.md#proposal
+[Metrics Stability Migration#General Migration Strategy]: 20190605-metrics-stability-migration.md#general-migration-strategy
+[Metrics Validation and Verification#Proposal]: 20190605-metrics-validation-and-verification.md#proposal
+[Metrics Stability to Beta#Proposal]: 20191028-metrics-stability-to-beta.md#proposal
 
 ## Design Details
 
@@ -103,8 +118,8 @@ See:
 1. [Metrics Stability Framework#Design Details]
 1. [Metrics Validation and Verification#Design Details]
 
-[Metrics Stability Framework#Design Details]: keps/sig-instrumentation/1209-metrics-stability/20190404-kubernetes-control-plane-metrics-stability.md#design-details
-[Metrics Validation and Verification#Design Details]: keps/sig-instrumentation/1209-metrics-stability/20190605-metrics-validation-and-verification.md#design-details
+[Metrics Stability Framework#Design Details]: 20190404-kubernetes-control-plane-metrics-stability.md#design-details
+[Metrics Validation and Verification#Design Details]: 20190605-metrics-validation-and-verification.md#design-details
 
 ### Graduation Criteria
 
@@ -115,8 +130,8 @@ See:
 1. [Metrics Stability Framework#Graduation Criteria]
 1. [Metrics Stability Migration#Graduation Criteria]
 
-[Metrics Stability Framework#Graduation Criteria]: keps/sig-instrumentation/1209-metrics-stability/20190404-kubernetes-control-plane-metrics-stability.md#graduation-criteria
-[Metrics Stability Migration#Graduation Criteria]: keps/sig-instrumentation/1209-metrics-stability/20190605-metrics-stability-migration.md#graduation-criteria
+[Metrics Stability Framework#Graduation Criteria]: 20190404-kubernetes-control-plane-metrics-stability.md#graduation-criteria
+[Metrics Stability Migration#Graduation Criteria]: 20190605-metrics-stability-migration.md#graduation-criteria
 
 #### Alpha -> Beta Graduation
 
@@ -125,36 +140,85 @@ See:
 1. [Metrics Validation and Verification#Graduation Criteria]
 1. [Metrics Stability to Beta#Graduation Criteria]
 
-[Metrics Validation and Verification#Graduation Criteria]: keps/sig-instrumentation/1209-metrics-stability/20190605-metrics-validation-and-verification.md#graduation-criteria
-[Metrics Stability to Beta#Graduation Criteria]: keps/sig-instrumentation/1209-metrics-stability/20191028-metrics-stability-to-beta.md#graduation-criteria
+[Metrics Validation and Verification#Graduation Criteria]: 20190605-metrics-validation-and-verification.md#graduation-criteria
+[Metrics Stability to Beta#Graduation Criteria]: 20191028-metrics-stability-to-beta.md#graduation-criteria
 
 #### Beta -> GA Graduation
 
-- Select stable metrics from control plane components
-- Implement the ability to turn off individual metrics (see [here](keps/sig-instrumentation/1209-metrics-stability/20191028-metrics-stability-to-beta.md#non-goals))
-
-**For non-optional features moving to GA, the graduation criteria must include 
-[conformance tests].**
-
-TODO(@logicalhan): ^
-
-[conformance tests]: https://git.k8s.io/community/contributors/devel/sig-architecture/conformance-tests.md
+- Metrics are now eligible to be promoted to STABLE status (we have some candidates in kube-apiserver).
+    - [apiserver_storage_object_counts](https://github.com/kubernetes/kubernetes/issues/98270)
+    - `apiserver_request_total` will also be promoted (as discussed in biweekly SIG apimachinery meeting)
+- Implement the ability to turn off individual metrics (see [here](20191028-metrics-stability-to-beta.md#non-goals))
+    - We need this because of stuff like this: [Unbounded valuesets for metric labels](https://github.com/kubernetes/kubernetes/issues/76302)
 
 ### Upgrade / Downgrade Strategy
 
 See:
 
-- [Deprecation Lifecycle](keps/sig-instrumentation/1209-metrics-stability/20190404-kubernetes-control-plane-metrics-stability.md#deprecation-lifecycle)
-- [Deprecation of modified metrics from metrics overhaul KEP](keps/sig-instrumentation/1209-metrics-stability/20190605-metrics-stability-migration.md#deprecation-of-modified-metrics-from-metrics-overhaul-kep)
-- [Escape Hatch](keps/sig-instrumentation/1209-metrics-stability/20191028-metrics-stability-to-beta.md#escape-hatch)
+- [Deprecation Lifecycle](20190404-kubernetes-control-plane-metrics-stability.md#deprecation-lifecycle)
+- [Deprecation of modified metrics from metrics overhaul KEP](20190605-metrics-stability-migration.md#deprecation-of-modified-metrics-from-metrics-overhaul-kep)
+- [Escape Hatch](20191028-metrics-stability-to-beta.md#escape-hatch)
+
+https://github.com/kubernetes/enhancements/blob/0f5bb1138a6dfd7f3d52fa901c2fba7abb7fb731/keps/sig-instrumentation/1209-metrics-stability/keps/sig-instrumentation/1209-metrics-stability/20190404-kubernetes-control-plane-metrics-stability.md#implementation-history
 
 ### Version Skew Strategy
 
 N/A
 
 ## Production Readiness Review Questionnaire
 
-N/A - this KEP predates PRR. @logicalhan to fill this in later if desired.
+#### How can this feature be enabled / disabled in a live cluster?
+
+The metrics stability framework adds developer tooling around commit pipelines and is not a user-facing feature per se. The part that is user-facing is the annotation on metrics with a stability level.
+
+This framework intends to increase reliability in control-plane management and so features in the metrics stability framework tend to 'fix' aspects of dev processes which lead to downstream breakages.
+
+Rollout, Upgrade and Rollback Planning
+This section must be completed when targeting beta graduation to a release.
+
+N/A, this isn't a feature per se.
+
+#### What specific metrics should inform a rollback?
+
+N/A
+
+### Monitoring Requirements
+
+#### How can an operator determine if the feature is in use by workloads?
+
+N/A
+
+#### What are the SLIs (Service Level Indicators) an operator can use to determine the health of the service?
+
+N/A
+
+#### Metrics
+
+The stability framework applies to all metrics which originate directly from the control-plane.
+
+### Dependencies
+
+This section must be completed when targeting beta graduation to a release.
+
+#### Does this feature depend on any specific services running in the cluster?
+
+N/A
+
+#### For GA, this section is required: approvers should be able to confirm the previous answers based on experience in the field.
+
+#### Will enabling / using this feature result in any new API calls? Describe them, providing:
+
+No.
+
+### Troubleshooting
+
+#### How does this feature react if the API server and/or etcd is unavailable?
+
+N/A (but if the component isn't available, no metrics are being scraped).
+
+#### What are other known failure modes?
+
+At worst, this thing can clog the commit pipeline (since it is effectively a conformance test for ensuring metric stability guarantees). In that case, we can simply turn off the verification and validation mechanism (i.e. the `hack/verify_generated_stable_metrics.sh` script) which effectively puts us back to where we were before the framework. Note that this basically allows developers to commit breaking changes to metrics and violate guarantees though.
 
 ## Implementation History
 
@@ -165,7 +229,7 @@ See:
 1. [Metrics Validation and Verification#Implementation History]
 1. [Metrics Stability to Beta#Implementation History]
 
-[Metrics Stability Framework#Implementation History]: keps/sig-instrumentation/1209-metrics-stability/20190404-kubernetes-control-plane-metrics-stability.md#implementation-history
-[Metrics Stability Migration#Implementation History]: keps/sig-instrumentation/1209-metrics-stability/20190605-metrics-stability-migration.md#implementation-history
-[Metrics Validation and Verification#Implementation History]: keps/sig-instrumentation/1209-metrics-stability/20190605-metrics-validation-and-verification.md#implementation-history
-[Metrics Stability to Beta#Implementation History]: keps/sig-instrumentation/1209-metrics-stability/20191028-metrics-stability-to-beta.md#implementation-history
+[Metrics Stability Framework#Implementation History]: 20190404-kubernetes-control-plane-metrics-stability.md#implementation-history
+[Metrics Stability Migration#Implementation History]: 20190605-metrics-stability-migration.md#implementation-history
+[Metrics Validation and Verification#Implementation History]: 20190605-metrics-validation-and-verification.md#implementation-history
+[Metrics Stability to Beta#Implementation History]: 20191028-metrics-stability-to-beta.md#implementation-history

keps/sig-instrumentation/1209-metrics-stability/kep.yaml

@@ -21,18 +21,19 @@ reviewers:
   - "@lavalamp" # api-machinery
   - "@dashpole" # node
   - "@ehashman" # instrumentation
-  - "@mml" 
+  - "@mml"
   - "@sttts" # api-machinery
   - "@bsalamat" # scheduling
   - "@andrewsykim" # cloud-provider
 approvers:
   - "@brancz"
 prr-approvers:
+  - "@johnbelamaric" # PRR-reviewer
 see-also:
   - "/keps/sig-instrumentation/1206-metrics-overhaul"
 
 # The target maturity stage in the current dev cycle for this KEP.
-stage: beta
+stage: stable
 
 # The most recent milestone for which work toward delivery of this KEP has been
 # done. This can be the current (upcoming) milestone, if it is being actively