Merge pull request kubernetes#2443 from andrewsykim/cloud-provider-feature-gates

KEP-2395 Introduce feature gates DisableCloudProviders and DisableKubeletCloudCredentialProvider

Author: k8s-ci-robot

keps/prod-readiness/sig-cloud-provider/2395.yaml

@@ -0,0 +1,3 @@
+kep-number: 2395
+alpha:
+  approver: "@wojtek-t"

keps/sig-cloud-provider/2395-removing-in-tree-cloud-providers/README.md

@@ -13,8 +13,16 @@
     - [Phase 1 - Moving Cloud Provider Code to Staging](#phase-1---moving-cloud-provider-code-to-staging)
     - [Phase 2 - Building CCM from Provider Repos](#phase-2---building-ccm-from-provider-repos)
     - [Phase 3 - Migrating Provider Code to Provider Repos](#phase-3---migrating-provider-code-to-provider-repos)
+    - [Phase 4 - Disabling In-Tree Providers](#phase-4---disabling-in-tree-providers)
   - [Staging Directory](#staging-directory)
     - [Cloud Provider Instances](#cloud-provider-instances)
+- [Production Readiness Review Questionnaire](#production-readiness-review-questionnaire)
+  - [Feature Enablement and Rollback](#feature-enablement-and-rollback)
+  - [Rollout, Upgrade and Rollback Planning](#rollout-upgrade-and-rollback-planning)
+  - [Monitoring Requirements](#monitoring-requirements)
+  - [Dependencies](#dependencies)
+  - [Scalability](#scalability)
+  - [Troubleshooting](#troubleshooting)
 - [Alternatives](#alternatives)
   - [Staging Alternatives](#staging-alternatives)
     - [Git Filter-Branch](#git-filter-branch)
@@ -99,10 +107,27 @@ The kube-controller-manager will still import the cloud provider implementations
 
 #### Phase 3 - Migrating Provider Code to Provider Repos
 
-In Phase 3, all code in `k8s.io/kubernetes/staging/src/k8s.io/legacy-cloud-providers/<provider>` will be removed and development of each cloud provider should be done in their respective external repos. It's important that by this phase, both in-tree and out-of-tree cloud providers are tested and production ready. Ideally most Kubernetes clusters in production should be using the out-of-tree provider before in-tree support is removed. A plan to migrate existing clusters from using the `kube-controller-manager` to the `cloud-controller-manager` is currently being developed. More details soon.
+In Phase 3, feature development is no longer accepted in `k8s.io/kubernetes/staging/src/k8s.io/legacy-cloud-providers/<provider>` and development of each cloud provider should be done in their respective external repos. Only bug and security fixes are accepted in-tree during this phase. It's important that by this phase, both in-tree and out-of-tree cloud providers are tested and production ready. Ideally most Kubernetes clusters in production should be using the out-of-tree provider before in-tree support is removed. A plan to migrate existing clusters from using the `kube-controller-manager` to the `cloud-controller-manager` is currently being developed. More details soon.
 
 External cloud providers can optionally still import providers from `k8s.io/legacy-cloud-providers` but no core components in `k8s.io/kubernetes` will import the legacy provider and the respective staging directory will be removed along with all its dependencies.
 
+#### Phase 4 - Disabling In-Tree Providers
+
+In Phase 4, two feature gates will be introduced to gradually disable and remove in-tree cloud providers:
+1. `DisableCloudProviders` - this feature gate will disable any functionality in kube-apiserver, kube-controller-manager and kubelet related to the `--cloud-provider` component flag.
+2. `DisableKubeletCloudCredentialProvider` - this feature gate will disable in-tree functionality in the kubelet to authenticate to the AWS, Azure and GCP container registries for image pull credentials.
+
+Both of these features gates only impacts functionality tied to the `--cloud-provider` flag, specifically in-tree volume plugins are not covered. Users should refer to CSI migration efforts for these.
+
+For alpha, the feature gates will be used for testing purposes. When enabled, tests will ensure that clusters with in-tree cloud providers disabled behaves as expected. This is targeted for v1.21 and will be
+disabled by default.
+
+For beta, the feature gates will be on by default, meaning core components will disallow use of in-tree cloud providers. This will act as a warning for users to migrate to external components. Users may
+choose to continue using the in-tree provider by explicitly disabling the feature gates. Beta is targeted for v1.23 or v1.24.
+
+For GA, the feature gate will be enabled by default and locked. Users at this point MUST migrate to external components and use of the in-tree cloud providers will be disallowed. One release after GA,
+the in-tree cloud providers can be safely removed. GA is targeted for v1.25 or v1.26.
+
 ### Staging Directory
 
 There are several sections of code which need to be shared between the K8s/K8s repo and the K8s/Cloud-provider repos.
@@ -169,6 +194,154 @@ import (
 )
 ```
 
+## Production Readiness Review Questionnaire
+
+### Feature Enablement and Rollback
+
+_This section must be completed when targeting alpha to a release._
+
+* **How can this feature be enabled / disabled in a live cluster?**
+  - [X] Feature gate (also fill in values in `kep.yaml`)
+    - Feature gate name: DisableCloudProviders
+    - Components depending on the feature gate: kubelet, kube-apiserver, kube-controller-manager
+  - [X] Feature gate (also fill in values in `kep.yaml`)
+    - Feature gate name: DisableKubeletCloudCredentialProvider
+    - Components depending on the feature gate: kubelet
+
+* **Does enabling the feature change any default behavior?**
+  Yes, enabling this feature will disable all capabilities enabled when `--cloud-provider` is set in core components.
+  Users need to ensure they have migrated to out-of-tree components prior to enabling this feature gate.
+  If appropriate extensions (CCM, credential provider, apiserver-network-proxy, etc) are in use, cloud provider capabilities
+  should remain the same at the very least.
+
+* **Can the feature be disabled once it has been enabled (i.e. can we roll back
+  the enablement)?**
+  Yes, the feature can be disabled once it is enabled. If disabled, users must ensure
+  that the CCM is no longer running in the cluster. Credential provider plugins and the
+  apiserver network proxy do not have to be stopped on rollback.
+
+* **What happens if we reenable the feature if it was previously rolled back?**
+
+  All capabilities from in-tree cloud providers will be re-disabled.
+
+* **Are there any tests for feature enablement/disablement?**
+  Adequate unit tests, component integration test and e2e tests will be added for this feature before
+  it is goes beta and on by default.
+
+### Rollout, Upgrade and Rollback Planning
+
+_This section must be completed when targeting beta graduation to a release._
+
+* **How can a rollout fail? Can it impact already running workloads?**
+
+  TBD for beta.
+
+* **What specific metrics should inform a rollback?**
+
+   TBD for beta.
+
+* **Were upgrade and rollback tested? Was the upgrade->downgrade->upgrade path tested?**
+
+  TBD for beta.
+
+* **Is the rollout accompanied by any deprecations and/or removals of features, APIs,
+fields of API types, flags, etc.?**
+
+  TBD for beta.
+
+### Monitoring Requirements
+
+_This section must be completed when targeting beta graduation to a release._
+
+* **How can an operator determine if the feature is in use by workloads?**
+
+  TBD for beta.
+
+* **What are the SLIs (Service Level Indicators) an operator can use to determine
+the health of the service?**
+
+  TBD for beta.
+
+* **What are the reasonable SLOs (Service Level Objectives) for the above SLIs?**
+
+  TBD for beta.
+
+* **Are there any missing metrics that would be useful to have to improve observability
+of this feature?**
+
+  TBD for beta.
+
+### Dependencies
+
+_This section must be completed when targeting beta graduation to a release._
+
+* **Does this feature depend on any specific services running in the cluster?**
+
+  TBD for beta.
+
+
+### Scalability
+
+_For alpha, this section is encouraged: reviewers should consider these questions
+and attempt to answer them._
+
+_For beta, this section is required: reviewers must answer these questions._
+
+_For GA, this section is required: approvers should be able to confirm the
+previous answers based on experience in the field._
+
+* **Will enabling / using this feature result in any new API calls?**
+
+  No, if anything it will result in reduced API calls in core components.
+
+* **Will enabling / using this feature result in introducing new API types?**
+
+  No.
+
+* **Will enabling / using this feature result in any new calls to the cloud
+provider?**
+
+  No, it will actually remove calls to the cloud provider in all core components.
+
+* **Will enabling / using this feature result in increasing size or count of
+the existing API objects?**
+
+  No.
+
+* **Will enabling / using this feature result in increasing time taken by any
+operations covered by [existing SLIs/SLOs]?**
+
+  No.
+
+* **Will enabling / using this feature result in non-negligible increase of
+resource usage (CPU, RAM, disk, IO, ...) in any components?**
+
+  No. In fact, it should reduce resource usage.
+
+### Troubleshooting
+
+The Troubleshooting section currently serves the `Playbook` role. We may consider
+splitting it into a dedicated `Playbook` document (potentially with some monitoring
+details). For now, we leave it here.
+
+_This section must be completed when targeting beta graduation to a release._
+
+* **How does this feature react if the API server and/or etcd is unavailable?**
+
+TBD for beta.
+
+* **What are other known failure modes?**
+
+TBD for beta.
+
+* **What steps should be taken if SLOs are not being met to determine the problem?**
+
+TBD for beta.
+
+[supported limits]: https://git.k8s.io/community//sig-scalability/configs-and-limits/thresholds.md
+[existing SLIs/SLOs]: https://git.k8s.io/community/sig-scalability/slos/slos.md#kubernetes-slisslos
+
+
 ## Alternatives
 
 ### Staging Alternatives

keps/sig-cloud-provider/2395-removing-in-tree-cloud-providers/kep.yaml

@@ -20,7 +20,29 @@ reviewers:
 approvers:
   - "@thockin"
   - "@liggit"
+prr-approvers:
+  - "@wojtek-t"
 editor: TBD
 creation-date: 2018-12-18
 last-updated: 2019-04-11
 status: implementable
+
+# The most recent milestone for which work toward delivery of this KEP has been
+# done. This can be the current (upcoming) milestone, if it is being actively
+# worked on.
+latest-milestone: "v1.21"
+
+stage: alpha
+
+# The following PRR answers are required at alpha release
+# List the feature gate name and the components for which it must be enabled
+feature-gates:
+  - name: DisableCloudProviders
+    components:
+      - kubelet
+      - kube-apiserver
+      - kube-controller-manager
+  - name: DisableKubeletCloudCredentialProviders
+    components:
+      - kubelet
+disable-supported: true