@@ -28,8 +28,8 @@ considerations/concerns apply.
28
28
29
29
## Implementation
30
30
31
- As mentioned above as a developer building systems on top of Cluster API, if you want to hook in the Cluster’s
32
- lifecycle via a Runtime Extension, you have to implement an HTTPS server handling a discovery request and a set
31
+ As mentioned above as a developer building systems on top of Cluster API, if you want to hook in the Cluster’s
32
+ lifecycle via a Runtime Extension, you have to implement an HTTPS server handling a discovery request and a set
33
33
of additional requests according to the OpenAPI specification for the Runtime Hook you are interested in.
34
34
35
35
The following shows a minimal example of a Runtime Extension server implementation:
@@ -56,7 +56,7 @@ import (
56
56
)
57
57
58
58
var (
59
- catalog = runtimecatalog.New ()
59
+ catalog = runtimecatalog.New ()
60
60
setupLog = ctrl.Log .WithName (" setup" )
61
61
62
62
// Flags.
@@ -165,36 +165,36 @@ func toPtr(f runtimehooksv1.FailurePolicy) *runtimehooksv1.FailurePolicy {
165
165
}
166
166
```
167
167
168
- For a full example see our [ test extension] ( https://github.com/kubernetes-sigs/cluster-api/tree/main/test/extension ) .
168
+ For a full example see our [ test extension] ( https://github.com/kubernetes-sigs/cluster-api/tree/main/test/extension ) .
169
169
170
- Please note that a Runtime Extension server can serve multiple Runtime Hooks (in the example above
171
- ` BeforeClusterCreate ` and ` BeforeClusterUpgrade ` ) at the same time. Each of them are handled at a different path, like the
170
+ Please note that a Runtime Extension server can serve multiple Runtime Hooks (in the example above
171
+ ` BeforeClusterCreate ` and ` BeforeClusterUpgrade ` ) at the same time. Each of them are handled at a different path, like the
172
172
Kubernetes API server does for different API resources. The exact format of those paths is handled by the server
173
173
automatically in accordance to the OpenAPI specification of the Runtime Hooks.
174
174
175
175
There is an additional ` Discovery ` endpoint which is automatically served by the ` Server ` . The ` Discovery ` endpoint
176
- returns a list of extension handlers to inform Cluster API which Runtime Hooks are implemented by this
176
+ returns a list of extension handlers to inform Cluster API which Runtime Hooks are implemented by this
177
177
Runtime Extension server.
178
178
179
179
Please note that Cluster API is only able to enforce the correct request and response types as defined by a Runtime Hook version.
180
180
Developers are fully responsible for all other elements of the design of a Runtime Extension implementation, including:
181
181
182
- - To choose which programming language to use; please note that Golang is the language of choice, and we are not planning
183
- to test or provide tooling and libraries for other languages. Nevertheless, given that we rely on Open API and plain
182
+ - To choose which programming language to use; please note that Golang is the language of choice, and we are not planning
183
+ to test or provide tooling and libraries for other languages. Nevertheless, given that we rely on Open API and plain
184
184
HTTPS calls, other languages should just work but support will be provided at best effort.
185
- - To choose if a dedicated or a shared HTTPS Server is used for the Runtime Extension (it can be e.g. also used to serve a
185
+ - To choose if a dedicated or a shared HTTPS Server is used for the Runtime Extension (it can be e.g. also used to serve a
186
186
metric endpoint).
187
187
188
- When using Golang the Runtime Extension developer can benefit from the following packages (provided by the
188
+ When using Golang the Runtime Extension developer can benefit from the following packages (provided by the
189
189
` sigs.k8s.io/cluster-api ` module) as shown in the example above:
190
190
191
- - ` exp/runtime/hooks/api/v1alpha1 ` contains the Runtime Hook Golang API types, which are also used to generate the
191
+ - ` exp/runtime/hooks/api/v1alpha1 ` contains the Runtime Hook Golang API types, which are also used to generate the
192
192
OpenAPI specification.
193
- - ` exp/runtime/catalog ` provides the ` Catalog ` object to register Runtime Hook definitions. The ` Catalog ` is then
194
- used by the ` server ` package to handle requests. ` Catalog ` is similar to the ` runtime.Scheme ` of the
193
+ - ` exp/runtime/catalog ` provides the ` Catalog ` object to register Runtime Hook definitions. The ` Catalog ` is then
194
+ used by the ` server ` package to handle requests. ` Catalog ` is similar to the ` runtime.Scheme ` of the
195
195
` k8s.io/apimachinery/pkg/runtime ` package, but it is designed to store Runtime Hook registrations.
196
196
- ` exp/runtime/server ` provides a ` Server ` object which makes it easy to implement a Runtime Extension server.
197
- The ` Server ` will automatically handle tasks like Marshalling/Unmarshalling requests and responses. A Runtime
197
+ The ` Server ` will automatically handle tasks like Marshalling/Unmarshalling requests and responses. A Runtime
198
198
Extension developer only has to implement a strongly typed function that contains the actual logic.
199
199
200
200
## Guidelines
@@ -203,23 +203,23 @@ While writing a Runtime Extension the following important guidelines must be con
203
203
204
204
### Timeouts
205
205
206
- Runtime Extension processing adds to reconcile durations of Cluster API controllers. They should respond to requests
207
- as quickly as possible, typically in milliseconds. Runtime Extension developers can decide how long the Cluster API Runtime
208
- should wait for a Runtime Extension to respond before treating the call as a failure (max is 30s) by returning the timeout
209
- during discovery. Of course a Runtime Extension can trigger long-running tasks in the background, but they shouldn't block
210
- synchronously.
206
+ Runtime Extension processing adds to reconcile durations of Cluster API controllers. They should respond to requests
207
+ as quickly as possible, typically in milliseconds. Runtime Extension developers can decide how long the Cluster API Runtime
208
+ should wait for a Runtime Extension to respond before treating the call as a failure (max is 30s) by returning the timeout
209
+ during discovery. Of course a Runtime Extension can trigger long-running tasks in the background, but they shouldn't block
210
+ synchronously.
211
211
212
212
### Availability
213
213
214
214
Runtime Extension failure could result in errors in handling the workload clusters lifecycle, and so the implementation
215
- should be robust, have proper error handling, avoid panics, etc.. . Failure policies can be set up to mitigate the
216
- negative impact of a Runtime Extension on the Cluster API Runtime, but this option can’t be used in all cases
215
+ should be robust, have proper error handling, avoid panics, etc.. . Failure policies can be set up to mitigate the
216
+ negative impact of a Runtime Extension on the Cluster API Runtime, but this option can’t be used in all cases
217
217
(see [ Error Management] ( #error-management ) ).
218
218
219
219
### Blocking Hooks
220
220
221
- A Runtime Hook can be defined as "blocking" - e.g. the ` BeforeClusterUpgrade ` hook allows a Runtime Extension
222
- to prevent the upgrade from starting. A Runtime Extension registered for the ` BeforeClusterUpgrade ` hook
221
+ A Runtime Hook can be defined as "blocking" - e.g. the ` BeforeClusterUpgrade ` hook allows a Runtime Extension
222
+ to prevent the upgrade from starting. A Runtime Extension registered for the ` BeforeClusterUpgrade ` hook
223
223
can block by returning a non-zero ` retryAfterSeconds ` value. Following consideration apply:
224
224
225
225
- The system might decide to retry the same Runtime Extension even before the ` retryAfterSeconds ` period expires,
@@ -230,57 +230,57 @@ can block by returning a non-zero `retryAfterSeconds` value. Following considera
230
230
- If there is more than one Runtime Extension registered for the same Runtime Hook and at least one returns
231
231
` retryAfterSeconds ` , all Runtime Extensions will be called again.
232
232
233
- Detailed description of what "blocking" means for each specific Runtime Hooks is documented case by case
233
+ Detailed description of what "blocking" means for each specific Runtime Hooks is documented case by case
234
234
in the hook-specific implementation documentation (e.g. [ Implementing Lifecycle Hook Runtime Extensions] ( ./implement-lifecycle-hooks.md#Definitions ) ).
235
235
236
236
### Side Effects
237
237
238
- It is recommended that Runtime Extensions should avoid side effects if possible, which means they should operate
239
- only on the content of the request sent to them, and not make out-of-band changes. If side effects are required,
238
+ It is recommended that Runtime Extensions should avoid side effects if possible, which means they should operate
239
+ only on the content of the request sent to them, and not make out-of-band changes. If side effects are required,
240
240
rules defined in the following sections apply.
241
241
242
242
### Idempotence
243
243
244
- An idempotent Runtime Extension is able to succeed even in case it has already been completed before (the Runtime
245
- Extension checks current state and changes it only if necessary). This is necessary because a Runtime Extension
246
- may be called many times after it already succeeded because other Runtime Extensions for the same hook may not
244
+ An idempotent Runtime Extension is able to succeed even in case it has already been completed before (the Runtime
245
+ Extension checks current state and changes it only if necessary). This is necessary because a Runtime Extension
246
+ may be called many times after it already succeeded because other Runtime Extensions for the same hook may not
247
247
succeed in the same reconcile.
248
248
249
- A practical example that explains why idempotence is relevant is the fact that extensions could be called more
249
+ A practical example that explains why idempotence is relevant is the fact that extensions could be called more
250
250
than once for the same lifecycle transition, e.g.
251
251
252
252
- Two Runtime Extensions are registered for the ` BeforeClusterUpgrade ` hook.
253
- - Before a Cluster upgrade is started both extensions are called, but one of them temporarily blocks the operation
253
+ - Before a Cluster upgrade is started both extensions are called, but one of them temporarily blocks the operation
254
254
by asking to retry after 30 seconds.
255
255
- After 30 seconds the system retries the lifecycle transition, and both extensions are called again to re-evaluate
256
256
if it is now possible to proceed with the Cluster upgrade.
257
257
258
258
### Avoid dependencies
259
259
260
- Each Runtime Extension should accomplish its task without depending on other Runtime Extensions. Introducing
261
- dependencies across Runtime Extensions makes the system fragile, and it is probably a consequence of poor
260
+ Each Runtime Extension should accomplish its task without depending on other Runtime Extensions. Introducing
261
+ dependencies across Runtime Extensions makes the system fragile, and it is probably a consequence of poor
262
262
"Separation of Concerns" between extensions.
263
263
264
264
### Deterministic result
265
265
266
266
A deterministic Runtime Extension is implemented in such a way that given the same input it will always return
267
267
the same output.
268
268
269
- Some Runtime Hooks, e.g. like external patches, might explicitly request for corresponding Runtime Extensions
270
- to support this property. But we encourage developers to follow this pattern more generally given that it fits
269
+ Some Runtime Hooks, e.g. like external patches, might explicitly request for corresponding Runtime Extensions
270
+ to support this property. But we encourage developers to follow this pattern more generally given that it fits
271
271
well with practices like unit testing and generally makes the entire system more predictable and easier to troubleshoot.
272
272
273
273
### Error Management
274
274
275
275
In case a Runtime Extension returns an error, the error will be handled according to the corresponding failure policy
276
276
defined in the response of the Discovery call.
277
277
278
- If the failure policy is ` Ignore ` the error is going to be recorded in the controller's logs, but the processing
279
- will continue. However we recognize that this failure policy cannot be used in most of the use cases because Runtime
280
- Extension implementers want to ensure that the task implemented by an extension is completed before continuing with
278
+ If the failure policy is ` Ignore ` the error is going to be recorded in the controller's logs, but the processing
279
+ will continue. However we recognize that this failure policy cannot be used in most of the use cases because Runtime
280
+ Extension implementers want to ensure that the task implemented by an extension is completed before continuing with
281
281
the cluster's lifecycle.
282
282
283
- If instead the failure policy is ` Fail ` the system will retry the operation until it passes. The following general
283
+ If instead the failure policy is ` Fail ` the system will retry the operation until it passes. The following general
284
284
considerations apply:
285
285
286
286
- It is the responsibility of Cluster API components to surface Runtime Extension errors using conditions.
@@ -289,5 +289,39 @@ considerations apply:
289
289
- If there is more than one Runtime Extension registered for the same Runtime Hook and at least one of them fails,
290
290
all the registered Runtime Extension will be retried. See [ Idempotence] ( #idempotence )
291
291
292
- Additional considerations about errors that apply only to a specific Runtime Hook will be documented in the hook-specific
292
+ Additional considerations about errors that apply only to a specific Runtime Hook will be documented in the hook-specific
293
293
implementation documentation.
294
+
295
+ ## Tips & tricks
296
+
297
+ After you implemented and deployed a Runtime Extension you can manually test it by sending HTTP requests.
298
+ This can be for example done via kubectl:
299
+
300
+ Via ` kubectl create --raw ` :
301
+
302
+ ``` bash
303
+ # Send a Discovery Request to the webhook-service in namespace default with protocol https on port 443:
304
+ kubectl create --raw ' /api/v1/namespaces/default/services/https:webhook-service:443/proxy/hooks.runtime.cluster.x-k8s.io/v1alpha1/discovery' \
305
+ -f <( echo ' {"apiVersion":"hooks.runtime.cluster.x-k8s.io/v1alpha1","kind":"DiscoveryRequest"}' ) | jq
306
+ ```
307
+
308
+ Via ` kubectl proxy ` and ` curl ` :
309
+
310
+ ``` bash
311
+ # Open a proxy with kubectl and then use curl to send the request
312
+ # # First terminal:
313
+ kubectl proxy
314
+ # # Second terminal:
315
+ curl -X ' POST' ' http://127.0.0.1:8001/api/v1/namespaces/default/services/https:webhook-service:443/proxy/hooks.runtime.cluster.x-k8s.io/v1alpha1/discovery' \
316
+ -d ' {"apiVersion":"hooks.runtime.cluster.x-k8s.io/v1alpha1","kind":"DiscoveryRequest"}' | jq
317
+ ```
318
+
319
+ For more details about the API of the Runtime Extensions please see <button onclick =" openSwaggerUI ()" >Swagger UI</button >.
320
+
321
+ <script >
322
+ // openSwaggerUI calculates the absolute URL of the RuntimeSDK YAML file and opens Swagger UI.
323
+ function openSwaggerUI () {
324
+ var schemaURL = new URL (" runtime-sdk-openapi.yaml" , document .baseURI ).href
325
+ window .open (" https://editor.swagger.io/?url=" + schemaURL)
326
+ }
327
+ </script >
0 commit comments