Merge pull request kubernetes#2405 from Jefftree/ssa

Update Server Side Apply KEP for 1.21 GA

Author: k8s-ci-robot

keps/prod-readiness/sig-api-machinery/555.yaml

@@ -0,0 +1,3 @@
+kep-number: 555
+stable:
+  approver: "@deads2k"

keps/sig-api-machinery/555-server-side-apply/README.md

@@ -19,6 +19,13 @@
       - [Proposed Change](#proposed-change)
       - [Alternatives](#alternatives)
       - [Implementation History](#implementation-history)
+- [Production Readiness Review Questionnaire](#production-readiness-review-questionnaire)
+  - [Feature Enablement and Rollback](#feature-enablement-and-rollback)
+  - [Rollout, Upgrade and Rollback Planning](#rollout-upgrade-and-rollback-planning)
+  - [Monitoring Requirements](#monitoring-requirements)
+  - [Dependencies](#dependencies)
+  - [Scalability](#scalability)
+  - [Troubleshooting](#troubleshooting)
   - [Risks and Mitigations](#risks-and-mitigations)
   - [Testing Plan](#testing-plan)
 - [Graduation Criteria](#graduation-criteria)
@@ -279,6 +286,176 @@ The conversion between the two and creating the diff was complex and would have
 
 - 12/2019 [#86083](https://github.com/kubernetes/kubernetes/pull/86083) implementing a poc for the described approach
 
+## Production Readiness Review Questionnaire
+
+<!--
+
+Production readiness reviews are intended to ensure that features merging into
+Kubernetes are observable, scalable and supportable; can be safely operated in
+production environments, and can be disabled or rolled back in the event they
+cause increased failures in production. See more in the PRR KEP at
+https://git.k8s.io/enhancements/keps/sig-architecture/1194-prod-readiness.
+
+The production readiness review questionnaire must be completed and approved
+for the KEP to move to `implementable` status and be included in the release.
+
+In some cases, the questions below should also have answers in `kep.yaml`. This
+is to enable automation to verify the presence of the review, and to reduce review
+burden and latency.
+
+The KEP must have a approver from the
+[`prod-readiness-approvers`](http://git.k8s.io/enhancements/OWNERS_ALIASES)
+team. Please reach out on the
+[#prod-readiness](https://kubernetes.slack.com/archives/CPNHUMN74) channel if
+you need any help or guidance.
+
+-->
+
+### Feature Enablement and Rollback
+
+_This section must be completed when targeting alpha to a release._
+
+* **How can this feature be enabled / disabled in a live cluster?**
+  - [x] Feature gate (also fill in values in `kep.yaml`)
+    - Feature gate name: [ServerSideApply](https://github.com/kubernetes/kubernetes/blob/master/staging/src/k8s.io/apiserver/pkg/features/kube_features.go#L100)
+    - Components depending on the feature gate: kube-apiserver
+
+* **Does enabling the feature change any default behavior?**
+
+  While this changes how objects are modified and then stored in the database, all the changes should be strictly backward compatible, and shouldn’t break existing automation or users. The increase in size can possibly have adverse, surprising consequences including increased memory usage for controllers, increased bandwidth usage when fetching objects, bigger objects when displaying for users (kubectl get -o yaml). We’re trying to mitigate all of these with the addition of a new header.
+
+* **Can the feature be disabled once it has been enabled (i.e. can we roll back
+  the enablement)?**
+  Also set `disable-supported` to `true` or `false` in `kep.yaml`.
+  Describe the consequences on existing workloads (e.g., if this is a runtime
+  feature, can it break the existing applications?).
+
+  Yes. The consequence is that managed fields will be reset for server-side applied objects (requiring a read/write cycle on the impacted resources).
+
+* **What happens if we reenable the feature if it was previously rolled back?**
+
+  The feature will be restored. Server-side applied objects will have lost their “set” which may cause some surprising behavior (fields might not be removed as expected).
+
+* **Are there any tests for feature enablement/disablement?**
+  The e2e framework does not currently support enabling or disabling feature
+  gates. However, unit tests in each component dealing with managing data, created
+  with and without the feature, are necessary. At the very least, think about
+  conversion tests if API types are being modified.
+
+  Tests are in place for upgrading from client side to server side apply and vice versa.
+
+### Rollout, Upgrade and Rollback Planning
+
+_This section must be completed when targeting beta graduation to a release._
+
+* **How can a rollout fail? Can it impact already running workloads?**
+  Try to be as paranoid as possible - e.g., what if some components will restart
+   mid-rollout?
+There is no specific way that the rollout can fail. The rollout can't impact existing workload.
+* **What specific metrics should inform a rollback?**
+
+  The feature shouldn't affect any existing behavior. A surprisingly high number of modification rejections could be a sign that something is not working properly.
+
+* **Were upgrade and rollback tested? Was the upgrade->downgrade->upgrade path tested?**
+
+  Because the feature doesn't affect existing behavior, rollback and upgrades haven't be specifically tested.
+The feature is being used by the cluster role aggregator though. Upgrading/downgrading/upgrading, which
+could result in the managedFields being removed, wouldn't cause any problems since the `Rules` field
+filled by the controller is `atomic`, and thus doesn't depend on the current state of the managedFields.
+
+  The new `managedFields` field is cleared when it is incorrect. That protects us from having invalid data inserted by a potential bad upgrade.
+
+* **Is the rollout accompanied by any deprecations and/or removals of features, APIs,
+fields of API types, flags, etc.?** No
+No.
+### Monitoring Requirements
+
+_This section must be completed when targeting beta graduation to a release._
+
+* **How can an operator determine if the feature is in use by workloads?**
+  Ideally, this should be a metric. Operations against the Kubernetes API (e.g.,
+  checking if there are objects with field X set) may be a last resort. Avoid
+  logs or events for this purpose.
+
+  Any existing metric split by request verb will record the [APPLY](https://github.com/kubernetes/kubernetes/blob/8f6ffb24df989608b87451f89b8ac9fc338ed71c/staging/src/k8s.io/apiserver/pkg/endpoints/metrics/metrics.go#L507-L509) verb if the feature is in use.
+
+  Additionally, the OpenAPI spec exposes the available media-type for each individual endpoint. The presence of the `apply` type for the PATCH verb of a endpoints indicates whether the feature is enabled for that specific resource, e.g.
+```json
+...
+"patch": {
+  "consumes": [
+     "application/json-patch+json",
+     "application/merge-patch+json",
+     "application/strategic-merge-patch+json",
+     "application/apply-patch+yaml"
+   ],
+    ...
+}
+...
+```
+
+* **What are the SLIs (Service Level Indicators) an operator can use to determine
+the health of the service?**
+
+  There is no specific metric attached to server side apply. All PATCH requests that utilize SSA will use the verb APPLY when logging metrics. API Server metrics that are split by verb automatically include this. They include `apiserver_request_total`, `apiserver_longrunning_gauge`, `apiserver_response_sizes`, `apiserver_request_terminations_total`, `apiserver_selfrequest_total`
+    - Components exposing the metric: kube-apiserver
+
+  Apply requests (`PATCH` with `application/apply-patch+yaml` mime type) have the same level of SLIs as other types of requests.
+
+* **What are the reasonable SLOs (Service Level Objectives) for the above SLIs?** n/a
+Apply requests (`PATCH` with `application/apply-patch+yaml` mime type) have the same level of SLOs as other types of requests.
+* **Are there any missing metrics that would be useful to have to improve observability
+of this feature?** n/a
+
+### Dependencies
+
+* **Does this feature depend on any specific services running in the cluster?** No
+
+### Scalability
+
+* **Will enabling / using this feature result in any new API calls?** No
+
+* **Will enabling / using this feature result in introducing new API types?**
+  Describe them, providing: No
+
+* **Will enabling / using this feature result in any new calls to the cloud
+provider?** No
+
+* **Will enabling / using this feature result in increasing size or count of
+the existing API objects?** Objects applied using server side apply will have their managed fields metadata populated. `managedFields` metadata fields can represent up to 60% of the total size of an object, increasing the size of objects.
+
+* **Will enabling / using this feature result in increasing time taken by any
+operations covered by [existing SLIs/SLOs]?** No
+
+* **Will enabling / using this feature result in non-negligible increase of
+resource usage (CPU, RAM, disk, IO, ...) in any components?** Since objects are larger with the new `managedFields`, caches as well as network bandwidth requirement will increase.
+
+### Troubleshooting
+
+The Troubleshooting section currently serves the `Playbook` role. We may consider
+splitting it into a dedicated `Playbook` document (potentially with some monitoring
+details). For now, we leave it here.
+
+_This section must be completed when targeting beta graduation to a release._
+
+* **How does this feature react if the API server and/or etcd is unavailable?**
+
+  The feature is part of of the API server and will not function without it
+
+* **What are other known failure modes?**
+  For each of them, fill in the following information by copying the below template:
+  - [Failure mode brief description]
+    - Detection: How can it be detected via metrics? Stated another way:
+      how can an operator troubleshoot without logging into a master or worker node? Apply requests (`PATCH` with `application/apply-patch+yaml` mime type) have the same level of SLIs as other types of requests.
+    - Mitigations: What can be done to stop the bleeding, especially for already
+      running user workloads? This shouldn't affect running workloads, and this feature shouldn't alter the behavior of previously existing mechanisms like PATCH and PUT.
+    - Diagnostics: What are the useful log messages and their required logging
+      levels that could help debug the issue? The feature uses very little logging, and errors should be returned directly to the user.
+      Not required until feature graduated to beta.
+    - Testing: Are there any tests for failure mode? Failure modes are tested exhaustively both as unit-tests and as integration tests.
+
+* **What steps should be taken if SLOs are not being met to determine the problem?** n/a
+
 ### Risks and Mitigations
 
 We used a feature branch to ensure that no partial state of this feature would
@@ -341,6 +518,8 @@ Integration tests for:
 - [x] Apply works with custom resources. [link](https://github.com/kubernetes/kubernetes/blob/b55417f429353e1109df8b3bfa2afc8dbd9f240b/staging/src/k8s.io/apiextensions-apiserver/test/integration/apply_test.go#L34-L117)
 - [x] Run kubectl apply tests with server-side flag enabled. [link](https://github.com/kubernetes/kubernetes/blob/81e6407393aa46f2695e71a015f93819f1df424c/test/cmd/apply.sh#L246-L314)
 
+E2E and Conformance tests will be added for GA.
+
 ## Graduation Criteria
 
 An alpha version of this is targeted for 1.14.
@@ -349,8 +528,11 @@ This can be promoted to beta when it is a drop-in replacement for the existing
 kubectl apply, and has no regressions (which aren't bug fixes). This KEP will be
 updated when we know the concrete things changing for beta.
 
-This will be promoted to GA once it's gone a sufficient amount of time as beta
-with no changes. A KEP update will precede this.
+A GA version of this is targeted for 1.21.
+
+- E2E tests are created and graduate to conformance
+- [Apply for client-go's typed client](https://github.com/kubernetes/enhancements/tree/master/keps/sig-api-machinery/2144-clientgo-apply) is implemented and at least one kube-controller-manager uses that client
+- Outstanding bugs around status wiping and scale subresource are fixed
 
 ### Upgrade / Downgrade Strategy
 
@@ -423,6 +605,7 @@ annotation is preserved and up-to-date as described in the downgrade above.
 * Early 2018: @lavalamp begins thinking about apply and writing design docs
 * 2018Q3: Design shift from merge + diff to tracking field managers.
 * 2019Q1: Alpha.
+* 2019Q3: Beta.
 
 (For more details, one can view the apply-wg recordings, or join the mailing list
 and view the meeting notes. TODO: links)

keps/sig-api-machinery/555-server-side-apply/kep.yaml

@@ -11,13 +11,29 @@ reviewers:
   - "@erictune"
 approvers:
   - "@bgrant0607"
+prr-approvers:
+  - "@deads2k"
 editor: TBD
 creation-date: 2018-03-28
-last-updated: 2018-03-28
+last-updated: 2021-02-21
 status: implementable
 see-also:
   - n/a
 replaces:
   - n/a
 superseded-by:
   - n/a
+
+stage: stable
+latest-milestone: "v1.21"
+
+milestone:
+  alpha: "v1.14"
+  beta: "v1.16"
+  stable: "v1.21"
+
+feature-gates:
+  - name: ServerSideApply
+    components:
+      - kube-apiserver
+disable-supported: true