Skip to content

Commit 09548f5

Browse files
k8s-ci-robotk8s-ci-robot
authored
Merge pull request kubernetes#2041 from mrunalp/cri_beta
KEP 2040: Kubelet CRI support
2 parents 18116e0 + 4c6a1a6 commit 09548f5

File tree

2 files changed

+304
-0
lines changed

2 files changed

+304
-0
lines changed

‎keps/sig-node/2040-kubelet-cri/README.md

Lines changed: 270 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,270 @@
1+
2+
# Kubelet CRI support
3+
4+
<!-- toc -->
5+
- [Release Signoff Checklist](#release-signoff-checklist)
6+
- [Summary](#summary)
7+
- [Motivation](#motivation)
8+
- [Goals](#goals)
9+
- [Non-Goals](#non-goals)
10+
- [Proposal](#proposal)
11+
- [Risks and Mitigations](#risks-and-mitigations)
12+
- [Design Details](#design-details)
13+
- [Identified Work Items](#identified-work-items)
14+
- [Changes from v1alpha2 to v1beta1](#changes-from-v1alpha2-to-v1beta1)
15+
- [Clean Up](#clean-up)
16+
- [Test Plan](#test-plan)
17+
- [Graduation Criteria](#graduation-criteria)
18+
- [Alpha -&gt; Beta Graduation](#alpha---beta-graduation)
19+
- [Beta -&gt; GA Graduation](#beta---ga-graduation)
20+
- [Upgrade / Downgrade Strategy](#upgrade--downgrade-strategy)
21+
- [Version Skew Strategy](#version-skew-strategy)
22+
- [Production Readiness Review Questionnaire](#production-readiness-review-questionnaire)
23+
- [Feature Enablement and Rollback](#feature-enablement-and-rollback)
24+
- [Rollout, Upgrade and Rollback Planning](#rollout-upgrade-and-rollback-planning)
25+
- [Monitoring Requirements](#monitoring-requirements)
26+
- [Dependencies](#dependencies)
27+
- [Scalability](#scalability)
28+
- [Troubleshooting](#troubleshooting)
29+
- [Implementation History](#implementation-history)
30+
<!-- /toc -->
31+
32+
## Release Signoff Checklist
33+
Items marked with (R) are required *prior to targeting to a milestone / release*.
34+
35+
- [x] (R) Enhancement issue in release milestone, which links to KEP dir in [kubernetes/enhancements] (not the initial KEP PR)
36+
- [x] (R) KEP approvers have approved the KEP status as `implementable`
37+
- [x] (R) Design details are appropriately documented
38+
- [x] (R) Test plan is in place, giving consideration to SIG Architecture and SIG Testing input
39+
- [x] (R) Graduation criteria is in place
40+
- [x] (R) Production readiness review completed
41+
- [ ] Production readiness review approved
42+
- [ ] "Implementation History" section is up-to-date for milestone
43+
- [ ] User-facing documentation has been created in [kubernetes/website], for publication to [kubernetes.io]
44+
- [ ] Supporting documentation—e.g., additional design documents, links to mailing list discussions/SIG meetings, relevant PRs/issues, release notes
45+
46+
[kubernetes.io]: https://kubernetes.io/
47+
[kubernetes/enhancements]: https://git.k8s.io/enhancements
48+
[kubernetes/kubernetes]: https://git.k8s.io/kubernetes
49+
[kubernetes/website]: https://git.k8s.io/website
50+
51+
## Summary
52+
Identify remaining gaps to promote CRI to Beta and GA to reflect its practical use in production for many years.
53+
54+
## Motivation
55+
CRI based runtimes such as CRI-O and containerd have been in use in production for over a year now with the current CRI API.
56+
We want to signal to the users that the CRI API is production ready and they should feel comfortable moving away dockershim
57+
as it is slated to be deprecated.
58+
59+
### Goals
60+
- Graduate the CRI API to stable.
61+
- Identify any fields that need to made more type safe such as Seccomp.
62+
- Address and cleanup the notes/todos in the CRI.
63+
64+
### Non-Goals
65+
- Block on any big new features.
66+
67+
## Proposal
68+
Evolve the CRI API version as we address feedback in each milestone towards stable.
69+
- v1alpha2 (alpha, current state)
70+
- v1beta (beta, proposed 1.20)
71+
- v1 (stable, TBD)
72+
73+
### Risks and Mitigations
74+
| Risk | Detail | Mitigation |
75+
|---|---|---|
76+
| CRI stats performance | CRI stats performance may be worse compared to cadvisor | Measure performance and share report with community |
77+
78+
## Design Details
79+
80+
### Identified Work Items
81+
- No longer map the `container-runtime-endpoint` flag as experimental.
82+
- Keep the `image-service-endpoint` flag as experimental and evaluate if it makes sense to keep
83+
as a configurable or remove it.
84+
85+
#### Changes from v1alpha2 to v1beta1
86+
87+
- kubenet: There exists an open TODO in the specification to remove support for setting PodCidr for kubenet networking. However for CRI
88+
implementations CNI is the existing standard and is primarily the only solution being tested with the CRI container runtime integrations.
89+
Need Sig-Node and Sig-Networking to help validate if / when kubenet is being deprecated and if we should deprecate this before beta. If not when.
90+
https://github.com/kubernetes/kubernetes/issues/62288
91+
type NetworkConfig struct {
92+
// CIDR to use for pod IP addresses. If the CIDR is empty, runtimes
93+
// should omit it.
94+
PodCidr string `protobuf:"bytes,1,opt,name=pod_cidr,json=podCidr,proto3" json:"pod_cidr,omitempty"`
95+
XXX_NoUnkeyedLiteral struct{} `json:"-"`
96+
XXX_sizecache int32 `json:"-"`
97+
}
98+
99+
#### Clean Up
100+
- Removal of TODOs that are no longer valid should be done before v1beta. We have scraped the api specification once and have a small
101+
list of commits to file.
102+
103+
### Test Plan
104+
- Review of the existing test cases in critest and adding more if we find any gaps.
105+
- Make sure we have e2e node (and possibly selected e2e conformance) tests running on more than one CRI implementation.
106+
107+
### Graduation Criteria
108+
109+
#### Alpha -> Beta Graduation
110+
111+
- Passes all existing CRI tests on at least two container runtimes (sig-node(e2e-node) and cri-tools(critest)).
112+
- Is in production on numerous clouds. (Note: this reflects the urgency of the signal to move off non CRI solutions.)
113+
- Documentation is updated to reflect beta status.
114+
115+
#### Beta -> GA Graduation
116+
- TBD
117+
118+
### Upgrade / Downgrade Strategy
119+
Kubelet and the runtime versions should use the same CRI version in lock-step.
120+
Upgrade involves draining all pods from a node, installing a CRI runtime with this version of the API and update to a matching kubelet
121+
and then make the node schedulable again.
122+
123+
### Version Skew Strategy
124+
Kubelet and the CRI runtime versions are expected to match so we don't have to worry about it for v1beta1.
125+
126+
## Production Readiness Review Questionnaire
127+
128+
### Feature Enablement and Rollback
129+
* **How can this feature be enabled / disabled in a live cluster?**
130+
- [ ] Feature gate (also fill in values in `kep.yaml`)
131+
- Feature gate name:
132+
- Components depending on the feature gate:
133+
- [x] Other
134+
- Describe the mechanism:
135+
Install, configure, and run a CRI runtime on a node. Change the kubelet configuration to point to the CRI runtime socket and restart the kubelet.
136+
- Will enabling / disabling the feature require downtime of the control
137+
plane?
138+
No. The control plane nodes could be modified one at a time to switch to CRI runtimes.
139+
- Will enabling / disabling the feature require downtime or reprovisioning of a node?
140+
Yes. One could re-provision an existing nodes or provision new nodes with a CRI runtime and kubelet configured to talk to that runtime and then
141+
migrate your existing workloads to the new nodes.
142+
143+
* **Does enabling the feature change any default behavior?**
144+
- It changes the default container runtime from dockershim, but the container workloads are expected to work the same way as they do with dockershim.
145+
146+
* **Can the feature be disabled once it has been enabled (i.e. can we roll back the enablement)?**
147+
Yes, the users could switch back to dockershim on a node reversing the process of installing CRI based runtime.
148+
149+
* **What happens if we reenable the feature if it was previously rolled back?**
150+
No impact per existing kubernetes policy for draining nodes for node lifecyle. IOW container runtime being used CRI or internal docker-shim is tied
151+
to node lifecycle operations.
152+
153+
* **Are there any tests for feature enablement/disablement?**
154+
No impact for v1 vs v2 or v1 point release extensions for the CRI api. Instead container runtimes will expose grpc service endpoints on a single socket
155+
for the CRI services as separate v1/v2 service types. A container runtime would have to provide two endpoints for each service by type if it wants to
156+
support two different versions of kubelet/runtime integrations.
157+
158+
### Rollout, Upgrade and Rollback Planning
159+
160+
_This section must be completed when targeting beta graduation to a release._
161+
162+
* **How can a rollout fail? Can it impact already running workloads?**
163+
Workloads scheduled on nodes with a CRI runtime may fail due to some misconfiguration of that node. Yes, it could impact running workloads
164+
since we depend upon draining and moving workloads around to switch to CRI runtimes on a node.
165+
166+
* **What specific metrics should inform a rollback?**
167+
- Nodes that have been switched to CRI runtime are not ready.
168+
- Workloads are failing to come up on a CRI configured node.
169+
170+
* **Were upgrade and rollback tested? Was the upgrade->downgrade->upgrade path tested?**
171+
We don't expect to do any automated upgrade or rollback from and to dockershim so this doesn't apply.
172+
173+
* **Is the rollout accompanied by any deprecations and/or removals of features, APIs,
174+
fields of API types, flags, etc.?**
175+
- TODO
176+
177+
### Monitoring Requirements
178+
179+
_This section must be completed when targeting beta graduation to a release._
180+
181+
* **How can an operator determine if the feature is in use by workloads?**
182+
The Node object returns the configured CRI runtime.
183+
184+
* **What are the SLIs (Service Level Indicators) an operator can use to determine
185+
the health of the service?**
186+
- [ ] Metrics
187+
- Metric name:
188+
- [Optional] Aggregation method:
189+
- Components exposing the metric:
190+
- [x] Other (treat as last resort)
191+
- Details:
192+
- Node Ready
193+
194+
* **What are the reasonable SLOs (Service Level Objectives) for the above SLIs?**
195+
TBD
196+
197+
* **Are there any missing metrics that would be useful to have to improve observability
198+
of this feature?**
199+
TBD
200+
201+
### Dependencies
202+
203+
_This section must be completed when targeting beta graduation to a release._
204+
205+
* **Does this feature depend on any specific services running in the cluster?**
206+
No.
207+
208+
### Scalability
209+
210+
_For beta, this section is required: reviewers must answer these questions._
211+
212+
* **Will enabling / using this feature result in any new API calls?**
213+
214+
Exec/attach/port forwarding go through the API server.
215+
216+
* **Will enabling / using this feature result in introducing new API types?**
217+
218+
No, new k8s API types besides seccomp changes in the CRI.
219+
220+
* **Will enabling / using this feature result in any new calls to the cloud provider?**
221+
222+
No.
223+
224+
* **Will enabling / using this feature result in increasing size or count of the existing API objects?**
225+
226+
No.
227+
228+
* **Will enabling / using this feature result in increasing time taken by any operations covered by [existing SLIs/SLOs]?**
229+
230+
No.
231+
232+
* **Will enabling / using this feature result in non-negligible increase of resource usage (CPU, RAM, disk, IO, ...) in any components?**
233+
234+
We have an open item to dive deeper into perf comparison of CRI stats vs. cadvisor and will update here once we have more data.
235+
236+
### Troubleshooting
237+
238+
- Troubleshooting for CRI integrations requires a good set of documentation be provided by kubernetes for interactions with pods,
239+
containers, kubelet, security profiles, crictl (cri-tools client), developer and test (e2e, node, and critest) guides, and each
240+
of the container runtimes. As such, while in v1Beta there should be workgroup(s) formed or issues tracked for reaching GA criteria
241+
over the beta to GA period.
242+
243+
* **How does this feature react if the API server, kubelet and/or etcd is unavailable?**
244+
- Open streams for Exec/Attach/Port forward that are forwarded by kubelet to API Server will likely timeout and close if
245+
API Server becomes unavailable.
246+
- CRI Runtimes are resiliant to kubelet loosing connection over GRPC CRI calls.
247+
- CRI Runtimes Integrations are not known to checkpoint using etcd and thus are not directly affected by etcd at the node.
248+
249+
250+
* **What are other known failure modes?**
251+
For each of them, fill in the following information by copying the below template:
252+
- [Failure mode brief description]
253+
- Detection: How can it be detected via metrics? Stated another way:
254+
how can an operator troubleshoot without logging into a master or worker node?
255+
- Mitigations: What can be done to stop the bleeding, especially for already
256+
running user workloads?
257+
- Diagnostics: What are the useful log messages and their required logging
258+
levels that could help debug the issue?
259+
Not required until feature graduated to beta.
260+
- Testing: Are there any tests for failure mode? If not, describe why.
261+
262+
* **What steps should be taken if SLOs are not being met to determine the problem?**
263+
264+
[supported limits]: https://git.k8s.io/community//sig-scalability/configs-and-limits/thresholds.md
265+
[existing SLIs/SLOs]: https://git.k8s.io/community/sig-scalability/slos/slos.md#kubernetes-slisslos
266+
267+
## Implementation History
268+
269+
- First version with v1alpha1 was released in k8s 1.5. See https://github.com/kubernetes/community/blob/ee783a18a34ef16da07f8d16d42782a6f78a9253/contributors/devel/sig-node/container-runtime-interface.md
270+
- v1alpha was released with k8s 1.10. See https://github.com/kubernetes/kubernetes/pull/58973

‎keps/sig-node/2040-kubelet-cri/kep.yaml

Lines changed: 34 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,34 @@
1+
title: Kubelet CRI support
2+
kep-number: 2040
3+
authors:
4+
- "@mrunalp"
5+
- "@mikebrow"
6+
owning-sig: sig-node
7+
status: implementable
8+
creation-date: 2020-09-30
9+
reviewers:
10+
- "@derekwaynecarr"
11+
- "@dchen1107"
12+
approvers:
13+
- "@derekwaynecarr"
14+
- "@dchen1107"
15+
prr-approvers:
16+
- "@deads2k"
17+
18+
# The target maturity stage in the current dev cycle for this KEP.
19+
stage: beta
20+
21+
# The most recent milestone for which work toward delivery of this KEP has been
22+
# done. This can be the current (upcoming) milestone, if it is being actively
23+
# worked on.
24+
latest-milestone: "v1.20"
25+
26+
# The milestone at which this feature was, or is targeted to be, at each stage.
27+
milestone:
28+
alpha: "v1.5"
29+
beta: "v1.20"
30+
stable: TBD
31+
32+
# The following PRR answers are required at beta release
33+
metrics:
34+
- TBD

0 commit comments

Comments
 (0)