You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
* Allow extension mechanisms (CustomResourceDefinitions, admission webhooks) to contribute warnings
90
89
91
90
## Proposal
92
91
@@ -98,6 +97,10 @@ When a deprecated API is used:
98
97
2. Increment a counter metric with labels for the API group, version, resource, API verb, and target removal major/minor version
99
98
3. Record an audit annotation indicating the request used a deprecated API
100
99
100
+
Allow custom resource definitions to indicate specific versions are deprecated
101
+
102
+
Allow admission webhooks to contribute warnings that are surfaced to the user
103
+
101
104
### Client-side changes
102
105
103
106
In client-go:
@@ -114,43 +117,114 @@ In kubectl, configure the per-process handler to:
114
117
115
118
1. dedupe warnings (only print a given warning once per invocation)
116
119
2. log to stderr with a `Warning:` prefix
117
-
3. color the `Warning:` prefix if stderr is a terminal and `$TERM != "dumb"` and `$NO_COLOR` is unset
120
+
3. color the `Warning:` prefix if the terminal is capable of displaying color
121
+
4. add a `--warnings-as-errors` option to exit with a non-zero exit code after executing the command if warnings were encountered
118
122
119
123
In kube-apiserver and kube-controller-manager, configure the process-wide handler to ignore warnings
120
124
121
125
## Design Details
122
126
123
-
Server-side:
127
+
### Server-side
128
+
129
+
**Warning mechanism:**
124
130
125
131
* Add a handler chain filter that attaches a WarningRecorder implementation to the request context
126
132
* Add a WarningRecorder implementation that de-duplicates the warning message per request and writes a `Warning` header
127
133
* the header structure is defined in [RFC2616#14.46](https://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.46)
128
-
* warnings would be written with code `299` and warn-agent of `-`
129
-
* In the endpoint installer, decorate handlers for deprecated APIs:
134
+
* warnings are written with code `299` and warn-agent of `-`
135
+
* no more than 4K of warning messages are written to avoid exceeding client and proxy header size limits;
136
+
if the total warning size exceeds that limit, individual warnings are truncated to 256 characters;
137
+
if the total warning size still exceeds that limit, warnings are output in the order added until the limit is reached;
138
+
for reference, the longest proposed deprecation warning generated by `kube-apiserver` would be ~170 characters,
139
+
so ~24 warning messages of that length could be accommodated
140
+
141
+
**In-process deprecated API warnings:**
142
+
143
+
In the endpoint installer, decorate handlers for deprecated APIs:
144
+
* add a warning header: `<group>/<version> <kind> is deprecated in v1.X+, unavailable in v1.Y+; use <replacementGroup>/<replacementVersion> <replacementKind>`
145
+
* increment the counter metric for the deprecated use
146
+
* add an audit annotation indicating the request was made to a deprecated API
147
+
148
+
**In-process validation and admission warnings:**
149
+
150
+
Enable adding warnings in validation, strategy, and admission handlers:
151
+
* Document when warnings should be surfaced (e.g. when a problematic value is present in a create request,
152
+
or is introduced by an update request)
153
+
* Define a helper method for adding a field-specific warning that takes a `field.Path` and message
154
+
155
+
**CustomResourceDefinition version deprecation warnings:**
156
+
157
+
Add `deprecated bool` and `deprecationWarning string` fields to the `CustomResourceDefinitionVersion` type:
158
+
159
+
```go
160
+
typeCustomResourceDefinitionVersionstruct {
161
+
...
162
+
// deprecated indicates this version of the custom resource API is deprecated.
163
+
// When set to true, API requests to this version receive a warning header in the server response,
164
+
// and an audit annotation indicating a deprecated API was used is added to the audit event for the request.
165
+
// Defaults to false.
166
+
// +optional
167
+
Deprecatedbool`json:"deprecated,omitempty"`
168
+
// deprecationWarning overrides the default warning returned to API clients.
169
+
// May only be set when `deprecated` is true.
170
+
// The default warning indicates this version is deprecated and recommends use
171
+
// of the newest served version of equal or greater stability, if one exists.
- Per-process / per-client warning handler precedence is honored
216
+
217
+
Deprecated API integration tests:
218
+
219
+
- Warning headers are returned when making requests to deprecated APIs
220
+
- Deprecated metrics are incremented when making requests to deprecated APIs
221
+
- Audit annotations are added when making requests to deprecated APIs
222
+
223
+
Extension mechanism integration tests:
224
+
225
+
- CustomResourceDefinition deprecated versions result in warning headers and audit annotations
226
+
- Admission webhook warnings are included in API responses to users
227
+
154
228
### Risks and Mitigations
155
229
156
230
**Metric cardinality**
@@ -169,6 +243,14 @@ Additional warning messages may be unexpected by kubectl or client-go consumers.
169
243
However, kubectl and client-go already output warning messages to stderr or via `klog.Warn`.
170
244
client-go consumers can programmatically modify or suppress the warning output at a per-process or per-client level.
171
245
246
+
**Header size**
247
+
248
+
The possibility of accumulating excessive numbers of warnings increases once extension mechanisms can contribute warnings.
249
+
This is mitigated by:
250
+
* Documented guidance on what types of issues should be surfaced as warnings
251
+
* Documented guidance on message length
252
+
* A warning recorder implementation that limits the individual and total length of warnings returned to a client
253
+
172
254
### Graduation Criteria
173
255
174
256
The structure of the `Warning` header is RFC-defined and unversioned.
@@ -182,18 +264,25 @@ drive automated action in clients, graduation criteria is primarily oriented
182
264
toward the stability level of the administrator metrics, and the ability to
183
265
disable the server sending warnings during the beta period.
184
266
185
-
#### Beta graduation
186
-
187
-
* Test plan is implemented
267
+
#### Beta
268
+
269
+
* Implement warning mechanism
270
+
* Implement in-process deprecated API warnings, metrics, audit annotations
271
+
* Complete test plan for implemented items
188
272
* API server output of `Warning` headers for deprecated API use is feature-gated and enabled by default
189
273
* The metric for deprecated API use is registered at [stability level `ALPHA`](https://github.com/kubernetes/enhancements/blob/master/keps/sig-instrumentation/20190404-kubernetes-control-plane-metrics-stability.md#stability-classes)
190
-
* client-go logs warnings by default
191
-
* kubectl outputs warnings to stderr
274
+
* client-go logs warnings with code `199` and `299`by default
275
+
* kubectl outputs warnings with code `199` and `299`to stderr by default
192
276
193
-
#### GA graduation
277
+
#### GA
194
278
195
279
* At least two releases after Beta
196
-
* Gathered feedback on metric structure and use from multi-cluster admins
280
+
* Gather feedback on metric structure and use from multi-cluster admins
281
+
* Implement in-process helpers for field-level validation warnings and admission warnings
* Custom resource and admission webhook documentation is extended to describe appropriate use of warnings
197
286
* API server output of `Warning` headers for deprecated API use is unconditionally enabled
198
287
* Server metric for deprecated API use is registered at [stability level `STABLE`](https://github.com/kubernetes/enhancements/blob/master/keps/sig-instrumentation/20190404-kubernetes-control-plane-metrics-stability.md#stability-classes)
199
288
@@ -208,6 +297,18 @@ Old clients making requests to a new API server ignore `Warning` headers.
208
297
209
298
New clients making requests to old API servers handle requests without `Warning` headers normally.
0 commit comments