You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
@@ -41,11 +47,15 @@ Items marked with (R) are required *prior to targeting to a milestone / release*
41
47
-[X] (R) KEP approvers have approved the KEP status as `implementable`
42
48
-[X] (R) Design details are appropriately documented
43
49
-[X] (R) Test plan is in place, giving consideration to SIG Architecture and SIG Testing input
50
+
-[X] e2e Tests for all Beta API Operations (endpoints)
51
+
-[X] (R) Ensure GA e2e tests meet requirements for [Conformance Tests](https://github.com/kubernetes/community/blob/master/contributors/devel/sig-architecture/conformance-tests.md)
52
+
-[X] (R) Minimum Two Week Window for GA e2e tests to prove flake free
44
53
-[X] (R) Graduation criteria is in place
54
+
-[X] (R) [all GA Endpoints](https://github.com/kubernetes/community/pull/1806) must be hit by [Conformance Tests](https://github.com/kubernetes/community/blob/master/contributors/devel/sig-architecture/conformance-tests.md)
45
55
-[X] (R) Production readiness review completed
46
56
-[X] (R) Production readiness review approved
47
57
-[X] "Implementation History" section is up-to-date for milestone
48
-
-~~ [] User-facing documentation has been created in [kubernetes/website], for publication to [kubernetes.io] ~~
58
+
-[X] User-facing documentation has been created in [kubernetes/website], for publication to [kubernetes.io]
49
59
-[X] Supporting documentation e.g., additional design documents, links to mailing list discussions/SIG meetings, relevant PRs/issues, release notes
50
60
51
61
[kubernetes.io]: https://kubernetes.io/
@@ -64,6 +74,10 @@ compute device allocation, thus, alongside the existing pod resources API endpoi
64
74
65
75
* Enable node monitoring agents to know the allocatable compute resources on a node, thus properly calculate the node compute resource utilization.
66
76
77
+
### Non-Goals
78
+
79
+
* Add new endpoint (like kubelet `/pods`)
80
+
67
81
## Proposal
68
82
69
83
### User Stories
@@ -172,6 +186,24 @@ the new proposed `GetAllocatableResources` API.
172
186
173
187
Add additional tests to prove that unhealthy devices are skipped as part of GetAllocatable and empty NUMA topology is not returned.
174
188
189
+
[X] I/we understand the owners of the involved components may require updates to
190
+
existing tests to make this code solid enough prior to committing the changes necessary
- Components depending on the feature gate: kubelet, 3rd party consumers.
253
+
254
+
###### Does enabling the feature change any default behavior?
216
255
217
-
***Does enabling the feature change any default behavior?** No
218
-
***Can the feature be disabled once it has been enabled (i.e. can we rollback the enablement)?** Yes, through feature gates.
219
-
***What happens if we reenable the feature if it was previously rolled back?** The API becomes available again. The API is stateless, so no recovery is needed, clients can just consume the data.
220
-
***Are there any tests for feature enablement/disablement?** A e2e test will demonstrate that when the feature gate is disabled, the API returns the appropriate error code.
256
+
No
257
+
258
+
###### Can the feature be disabled once it has been enabled (i.e. can we roll back the enablement)?
259
+
260
+
Yes, through feature gate. Once GA, the feature can't be disabled and is always enabled.
261
+
262
+
###### What happens if we reenable the feature if it was previously rolled back?
263
+
264
+
The API becomes available again. The API is stateless, so no recovery is needed, clients can just consume the data.
265
+
266
+
###### Are there any tests for feature enablement/disablement?
267
+
268
+
An e2e test will demonstrate that when the feature gate is disabled, the API returns the appropriate error code.
221
269
222
270
### Rollout, Upgrade and Rollback Planning
223
271
224
-
***How can a rollout fail? Can it impact already running workloads?** Kubelet may fail to start. The new API may report inconsistent data, or may cause the kubelet to crash.
225
-
***What specific metrics should inform a rollback?**`pod_resources_endpoint_errors_get_allocatable` - but only with feature gate enabled. Otherwise the API will always return a known error, giving a false negative signal.
226
-
***Were upgrade and rollback tested? Was upgrade->downgrade->upgrade path tested?** Not Applicable.
227
-
***Is the rollout accompanied by any deprecations and/or removals of features, APIs, fields of API types, flags, etc.?** No.
272
+
###### How can a rollout or rollback fail? Can it impact already running workloads?
273
+
274
+
Kubelet may fail to start. The new API may report inconsistent data, or may cause the kubelet to crash.
275
+
276
+
###### What specific metrics should inform a rollback?
277
+
278
+
`pod_resources_endpoint_errors_get_allocatable` - but only with feature gate enabled. Otherwise the API will always return a known error, giving a false negative signal.
279
+
280
+
###### Were upgrade and rollback tested? Was the upgrade->downgrade->upgrade path tested?
281
+
282
+
Not Applicable.
283
+
284
+
###### Is the rollout accompanied by any deprecations and/or removals of features, APIs, fields of API types, flags, etc.?
285
+
286
+
No (Not applicable)
228
287
229
-
### Monitoring requirements
230
-
***How can an operator determine if the feature is in use by workloads?**
231
-
- Look at the `pod_resources_endpoint_requests_get_allocatable` metric exposed by the kubelet.
232
-
- Clients are connected to the podresources unix socket, for example bychecking which containers mount the podresources socket path.
233
-
***What are the SLIs (Service Level Indicators) an operator can use to determine the health of the service?**
***What are the reasonable SLOs (Service Level Objectives) for the above SLIs?** N/A.
239
-
***Are there any missing metrics that would be useful to have to improve observability if this feature?** As part of this feature enhancement, per-API-endpoint resources metrics are being added; to observe this feature the `pod_resources_endpoint_requests_get_allocatable` metric should be used. We will also add error counting metrics to improve the observability of the API.
290
+
###### How can an operator determine if the feature is in use by workloads?
240
291
292
+
- Look at the `pod_resources_endpoint_requests_get_allocatable` metric exposed by the kubelet.
293
+
- Clients are connected to the podresources unix socket, for example by checking which containers mount the podresources socket path.
294
+
295
+
###### How can someone using this feature know that it is working for their instance?
296
+
297
+
-[ ] Events
298
+
- Event Reason:
299
+
-[ ] API .status
300
+
- Condition name:
301
+
- Other field:
302
+
-[X] Other (treat as last resort)
303
+
- Look at the `pod_resources_endpoint_requests_get_allocatable` and `pod_resources_endpoint_errors_get_allocatable` metrics exposed by the kubelet.
304
+
305
+
###### What are the reasonable SLOs (Service Level Objectives) for the enhancement?
306
+
307
+
100% in normal operation. The proposed API exposes in read only mode kubelet internal data, critical for functioning of the kubelet.
308
+
This data has to be available 100% of the time for the proper functioning of the kubelet, thus is expected to be available 100% of time.
309
+
The only possible error source is the API calls being throttled by the rate-limiting introduced with the GA graduation of the parent KEP 606.
310
+
311
+
###### What are the SLIs (Service Level Indicators) an operator can use to determine the health of the service?
###### Are there any missing metrics that would be useful to have to improve observability of this feature?
320
+
321
+
As part of this feature enhancement, per-API-endpoint resources metrics are being added; to observe this feature the `pod_resources_endpoint_requests_get_allocatable` metric should be used.
322
+
We added the `pod_resources_endpoint_errors_get_allocatable` metric to report errors. Because the nature of the API (exposing data already used by the kubelet with minimal processing)
323
+
the error counter is expected to be stable zero.
241
324
242
325
### Dependencies
243
326
244
-
***Does this feature depend on any specific services running in the cluster?** Not applicable.
327
+
###### Does this feature depend on any specific services running in the cluster?
328
+
329
+
No
245
330
246
331
### Scalability
247
332
248
-
***Will enabling / using this feature result in any new API calls?** No.
249
-
***Will enabling / using this feature result in introducing new API types?** No.
250
-
***Will enabling / using this feature result in any new calls to cloud provider?** No.
251
-
***Will enabling / using this feature result in increasing size or count of the existing API objects?** No.
252
-
***Will enabling / using this feature result in increasing time taken by any operations covered by [existing SLIs/SLOs][]?** No. Feature is out of existing any paths in kubelet.
253
-
***Will enabling / using this feature result in non-negligible increase of resource usage (CPU, RAM, disk, IO, ...) in any components?** DDOSing the API can lead to resource exhaustion. It is planned to be addressed as part of G.A.
254
-
Feature only collects data when requests comes in, data is then garbage collected. Data collected is proportional to the number of pods on the node.
333
+
###### Will enabling / using this feature result in any new API calls?
334
+
335
+
No
336
+
337
+
###### Will enabling / using this feature result in introducing new API types?
338
+
339
+
No
340
+
341
+
###### Will enabling / using this feature result in any new calls to the cloud provider?
342
+
343
+
No
344
+
345
+
###### Will enabling / using this feature result in increasing size or count of the existing API objects?
346
+
347
+
No
348
+
349
+
###### Will enabling / using this feature result in increasing time taken by any operations covered by existing SLIs/SLOs?
350
+
351
+
No. The feature is not affecting hot code paths in the kubelet, and just give access to cached data already computed by the kubelet for internal bookkeeping.
352
+
353
+
###### Will enabling / using this feature result in non-negligible increase of resource usage (CPU, RAM, disk, IO, ...) in any components?
354
+
355
+
Negligible amount of CPU and memory, because the endpoint queries existing data structures inside the kubelet.
356
+
357
+
###### Can enabling / using this feature result in resource exhaustion of some node resources (PIDs, sockets, inodes, etc.)?
358
+
359
+
No, because the endpoint queries existing data structures inside the kubelet.
255
360
256
361
### Troubleshooting
257
362
258
-
***How does this feature react if the API server and/or etcd is unavailable?**: No effect.
259
-
***What are other known failure modes?** feature gate disabled: the API will always return a well-known error. In normal operation, the API is expected to never return error and always return a valid response, because it utilizes internal kubelet data which is always available. Bugs may lead to the API to return unexpected errors, or to return inconsistent data. Consumers of the API should treat unexpected errors as bugs of this API.
260
-
***What steps should be taken if SLOs are not being met to determine the problem?** N/A
363
+
###### How does this feature react if the API server and/or etcd is unavailable?
364
+
365
+
No impact, the feature is node-local
366
+
367
+
###### What are other known failure modes?
368
+
369
+
feature gate disabled: the API will always return a well-known error. In normal operation, the API is expected to never return error and always return
370
+
a valid response, because it utilizes internal kubelet data which is always available.
371
+
Bugs may lead to the API to return unexpected errors, or to return inconsistent data.
372
+
Consumers of the API should treat unexpected errors as bugs of this API.
373
+
374
+
###### What steps should be taken if SLOs are not being met to determine the problem?
0 commit comments