Skip to content

Commit a1e7aa1

Browse files
kate-goldenringkate-goldenring
committed
SKIP 00X: Configuring Runtime Options in the Spin Containerd Shim
Signed-off-by: Kate Goldenring <[email protected]>
1 parent d74b448 commit a1e7aa1

File tree

1 file changed

+235
-0
lines changed

1 file changed

+235
-0
lines changed

‎containerd-shim-spin-config.md‎

Lines changed: 235 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,235 @@
1+
# SKIP 00X - Configuring Runtime Options in the `containerd-shim-spin`
2+
3+
Summary: Configuring runtime options in the `containerd-shim-spin`
4+
5+
Owner: Kate Goldenring <[email protected]>
6+
7+
Impacted Projects:
8+
9+
- [x] spin-operator
10+
- [ ] `spin kube` plugin
11+
- [x] runtime-class-manager
12+
- [x] containerd-shim-spin
13+
- [ ] Governance
14+
- [ ] Creates a new project
15+
16+
Created: 30/05/2024
17+
18+
## Overview
19+
20+
The `containerd-shim-spin` SpinKube executor currently lackd the ability to configure runtime flags for Spin applications. This proposal aims to address this by enabling three types of configurations: Spin execution settings, shim execution settings, and shim execution environment variables. These configurations will be set via Custom Resource Definitions (CRDs) in Kubernetes, specifically extending the SpinExecutor and SpinApp CRDs. This approach will allow operators to easily configure both the application and the platform execution environments.
21+
22+
## Background
23+
24+
SpinKube executors expose various options and arguments in order to configure how an application is executed. Take, for example, the following command:
25+
26+
```sh
27+
spin up --listen "127.0.0.1:3000" --disable-pooling --log-dir /var/logs --runtime-config-file /var/spin/config.toml
28+
```
29+
30+
The previous command tells the Spin runtime to do the following:
31+
32+
- Listen for requests to the Spin app on IP address and port `127.0.0.1:3000`
33+
- Disable Wasmtime's pooling instance allocator
34+
- Use `/var/logs` as the log directory for the stdout and stderr of components
35+
- Use `/var/spin/config.toml` as the runtime configuration file.
36+
37+
Currently, there is no way for configure any of these runtime flags through the `containerd-shim-spin`.
38+
39+
Operators may also wish to configure Wasm runtime environment variables, such as specifying that the traces of a Spin app be emitted to a specific exporter address (analogous to `OTEL_EXPORTER_OTLP_ENDPOINT=http://127.0.0.1:4318 spin up`).
40+
41+
A Runwasi containerd shim does more than execute a Wasm runtime such as Spin or Wasmtime or WasmEdge. It also pulls and prepares Wasm applications. The Shim may also have extra settings it would like to expose to configure how it prepares a Spin application. For example, a user should be able to disable pre-compilation for an application. The environment, such as the `RUST_LOG` level, of the shim should be configurable, too.
42+
43+
In summary, the following are types of configuration that should be supported:
44+
45+
- Wasm runtime specific options (oftentimes in the form of CLI flags: `spin up --listen "127.0.0.1:3000"`)
46+
- Wasm runtime environment variables (such as `OTEL_EXPORTER_OTLP_ENDPOINT=http://127.0.0.1:4318 spin up` )
47+
- Shim execution options (such as disabling pre-compilation)
48+
- Shim execution environment variables (such as `RUST LOG`)
49+
50+
Combining the first two categories and focusing on the Spin runtime, the list can be simplified:
51+
52+
- Spin execution configuration
53+
- Shim execution configuration
54+
- Shim execution environment variables
55+
56+
### Current Workarounds
57+
58+
#### Wasm Runtime Specific Options Workaround
59+
60+
None, unless an environment variable is also exposed for the flag and can be configured with the [Wasm runtime environment variables workaround](#Wasm-runtime-environment-variables-workaround).
61+
62+
These flags could be set in the runtime arguments for the container, such as the following in a Kubernetes PodSpec:
63+
64+
```sh
65+
containers:
66+
- name: spin-test
67+
image: ghcr.io/deislabs/containerd-wasm-shims/examples/spin-rust-hello:v0.10.0
68+
command: ["/"]
69+
args: ["--listen", "0.0.0.0:3000"]
70+
```
71+
72+
While the shim currently has access to these arguments from the [`RuntimeContext` args](https://github.com/containerd/runwasi/blob/f5f497c4b21a5d55613095a3ae878c4ef4b83b91/crates/containerd-shim-wasm/src/container/context.rs#L14), the shim does not parse these arguments and would need to be enlightened in a similar manner as the Spin CLI to discern which are for which trigger.
73+
74+
#### Wasm Runtime Environment Variables Workaround
75+
76+
There is no SpinKube native way to configure environment variables on the runtime process. Working around the Spin operator, these environment variables can be configured by setting them in the container spec of a Spin app deployment:
77+
78+
```yaml
79+
containers:
80+
- name: foo
81+
env:
82+
- name: NODE_IP
83+
valueFrom:
84+
fieldRef:
85+
fieldPath: status.hostIP
86+
- name: OTEL_EXPORTER_OTLP_ENDPOINT
87+
value: "http://$(NODE_IP):4318"
88+
- name: FOO
89+
value: "bar"
90+
```
91+
92+
#### Shim Execution Environment Variables Workaround
93+
94+
Containerd shims inherit the containerd environment variables. Therefore, a shim environment variable can be configured by setting it on the containerd process. For example, to configure the `RUST_LOG` level:
95+
96+
```sh
97+
RUST_LOG=info containerd --address /tmp/containerd.sock
98+
```
99+
100+
If the process is managed by systemd, you can add an environment variable to the systemd service file. First, stop the service and bring up the service file:
101+
102+
```sh
103+
systemctl stop containerd
104+
systemctl edit containerd
105+
```
106+
107+
At the top of the file, add the environment variable to the service:
108+
109+
```sh
110+
[Service]
111+
Environment="RUST_LOG=info"
112+
```
113+
114+
Finally, restart the containerd service:
115+
116+
```sh
117+
systemctl restart containerd
118+
```
119+
120+
> note, for k3s, edit the k3s service (`systemctl edit k3s`)
121+
122+
## Proposal
123+
124+
As laid out in the [background](#background), this SKIP aims to enable the following categories of configuration:
125+
126+
1. Spin execution configuration
127+
2. Shim execution configuration
128+
3. Shim execution environment variables
129+
130+
The first category discusses how to configure an app's execution, while the later two discuss how to configure the platform.
131+
132+
### User Interface
133+
134+
How does a user set this configuration? As is standard with Kubernetes, configuration should be set in Custom Resource Definitions. Rather than creating a new CRD for configuration, the `SpinExecutor` and `SpinApp` CRDs can be extended to support configuring the platform and app execution, respectively. Users should be able to configure Spin execution next to their app configuration in the `SpinApp` CRD. On the other hand, platform configuration is not done on a per app basis. It could be configured in the `SpinAppExecutor` CRD. This would help users easily see how the platform is configured for the executor their app is using. It would also enable creating multiple executors with distinct configuration.
135+
136+
### Configuring Spin Execution in Container Environment Variables
137+
138+
All Spin execution configuration will be defined as known environment variables with an expected range of values. All trigger configuration keys have the prefix "SPIN_<TRIGGER NAME>". Each executor will define which environment variables it supports. The following is a subset of configuration values that may be supported:
139+
140+
| Key | Spin CLI | Example Value|
141+
| ---- | ---- | ---- |
142+
| SPIN_HTTP_LISTEN_ADDR | `spin up --listen` | "0.0.0.0:3000" |
143+
| OTEL_EXPORTER_OTLP_ENDPOINT | `OTEL_EXPORTER_OTLP_ENDPOINT=http://123.4.5.6:4318" spin up` | "http://123.4.5.6:4318" |
144+
| SPIN_LOG_DIR | `spin up --log-dir /tmp/log` | "/tmp/log" |
145+
| RUNTIME_CONFIG_FILE | `spin up --runtime-config-file /var/config.toml` | "/var/config.toml" |
146+
147+
These values will be configured as environment variables in the Container spec. At first, this feels unadvisable, since it differs from user's previous understanding of Linux container environment variables. Deploying a Spin app shouldn't be compared to deploying a Linux container. A Linux container has one virtualized environment and listener. In a Linux container context, environment variables are set in the app execution and affect the business logic. On the other hand, a Spin app can have N many triggers and components, each of which has its own virtualized environment. While Spin does support setting environment variables in components, they are static. We have never encouraged the use of `spin up --env`, as it is more of a developer shortcut, since it sets the same set of environment variables in all components of an app. This makes each sandbox bigger than necessary and doesn't emphasize the component model's strengths.
148+
149+
That being said, we should steer Spin developers towards using configuration variables, which are already supported in the `SpinApp` CRD. In documentation, we should clarify the distinction between a Spin app and a Linux container, being sure to specify that container environment variables configure the Spin runtime with environment variables and that Spin Wasm components can only be configured with Spin configuration variables.
150+
151+
Each of these environment variables should be configurable from the SpinApp CRD under `SpinApp.spec.runtimeEnvironment`
152+
153+
```sh
154+
apiVersion: core.spinoperator.dev/v1alpha1
155+
kind: SpinApp
156+
metadata:
157+
name: myspinapp
158+
spec:
159+
image: "ghcr.io/myuser:v1"
160+
replicas: 1
161+
executor: containerd-shim-spin
162+
runtimeEnvironment:
163+
- name: SPIN_HTTP_LISTEN_ADDR
164+
value: "0.0.0.0:3000"
165+
```
166+
167+
> Note: `runtimeEnvironment` is distinct from `runtimeConfig`
168+
169+
The Spin Operator will then add each key-value pair to the Deployment of the SpinApp:
170+
171+
```sh
172+
spec
173+
containers:
174+
- name: myspinapp
175+
image: "ghcr.io/myuser:v1"
176+
env:
177+
- name: SPIN_HTTP_LISTEN_ADDR
178+
value: "0.0.0.0:3000"
179+
...
180+
```
181+
182+
The `containerd-shim-spin` instance will look for all supported runtime environment variables and configure execution accordingly.
183+
184+
### Configuring Shim Execution in Containerd Config Options
185+
186+
Containerd supports [configuring options for runtimes](https://github.com/containerd/containerd/blob/main/docs/man/containerd-config.toml.5.md#multiple-runtimes), such as the Spin shim. These options are set in the containerd configuration file (`config.toml`), which is where the runtime class manager notified containerd of the existence of the shim. The Runtime Class Manager could also add shim options when setting up the shim. For example, it may make the following additions to a containerd configuration file:
187+
188+
```toml
189+
# Tell containerd where to find the shim (it parses this as /containerd/path/containerd-shim-spin-v2)
190+
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes."spin"]
191+
runtime_type = "io.containerd.spin.v2"
192+
# Add options to each execution of the shim
193+
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes."spin".options]
194+
Foo = "bar"
195+
DISABLE_PRECOMPILATION = "true"
196+
```
197+
198+
> For an example implementation, see [gVisor's containerd runtime options configuration documentation](https://gvisor.dev/docs/user_guide/containerd/configuration/).
199+
200+
When a container is executed with CRI, these options are sent to Runwasi in a [`CreateTaskRequest`](https://github.com/containerd/containerd/blob/main/core/runtime/v2/README.md#container-level-shim-configuration). However, right now Runwasi does not parse them. Runwasi should parse the options,and pass them to the shim `Engine` through a new `configure_options` method that should run prior to any other methods. Runwasi may also extract it's own configuration from these containerd runtime options.
201+
202+
The following are Spin shim and Runwasi options that should be configurable:
203+
204+
| Key | Example Value | Scope |
205+
| ---- | ---- | ---- |
206+
| DISABLE_PRECOMPILATION | "true" | shim |
207+
| GRPC_MAX_WRITE_CHUNK_SIZE_BYTES | "15728640" | [Runwasi](https://github.com/containerd/runwasi/blob/7cc5e5a8025f1508c264dee1faf1c8c57cf645a1/crates/containerd-shim-wasm/src/sandbox/containerd/client.rs#L38) |
208+
209+
Ideally, these options could be set in the `SpinAppExecutor` CRD. Then, the RuntimeClassManager can watch the resource and update and restart containerd as needed.
210+
211+
### Configuring Shim Environment Variables via Containerd Environment
212+
213+
Any environment variables that need to be set on the shim process will need to continue to be set on the containerd process as described in the [Shim execution environment variables](#shim-execution-environment-variables).
214+
215+
## References
216+
217+
- Runwasi issue for tracking containerd config options for shims: https://github.com/containerd/runwasi/issues/573
218+
- `containerd-shim-spin` issue tracking inability to run two Spin apps in the same container due to fixed listen address: https://github.com/spinkube/containerd-shim-spin/issues/52
219+
- `containerd-shim-spin` issue considering which `spin up` flags to support in the shim: https://github.com/spinkube/containerd-shim-spin/issues/130
220+
221+
## Alternatives Considered
222+
223+
### Configuring Spin Execution Alternatives
224+
225+
The following are alternate strategies for configuring the Spin runtime in the `containerd-shim-spin`.
226+
227+
1. Spin execution options could be configured in in Pod annotations; however, it is not granular enough for the case of two Spin apps in a Pod, for example, [listen ports could still conflict](https://github.com/spinkube/containerd-shim-spin/issues/52).
228+
229+
2. As shown in the [runtime configuration workaround](#wasm-runtime-specific-options-workaround), options could be set as container arguments, but setting flags in CRDs is less ergonomic than environment variables, and parsing logic would need to be added to the shim. However, taking this path would potentially enable reusing Spin's reconciling of the mutual exclusiveness of flags.
230+
231+
3. Spin Runtime config (`runtime-config.toml`) could be enhanced to support configuring triggers. Right now, runtime config is used to configure host components. So far, it has been used the describe how to configure the host environment for the guest. That could encompass considering how the host should be triggered to invoke the guest. This may be the better long term solution; however, it will take design to consider the scope of Spin runtime config.
232+
233+
4. Alternatively, there may be places each of these configuration options fit outside of CLI flags. For example, the port for HTTP triggered apps could be configured in the trigger specification in the `spin.toml`. Log directory is already a part of [runtime config](https://github.com/fermyon/spin/blob/b3db535c9edb72278d4db3a201f0ed214e561354/crates/trigger/src/runtime_config.rs#L203-L204) despite the fact that users are more comfortable configuring it through the Spin CLI.
234+
235+
5. Having an extensible way to configure trigger execution will be important to trigger developers. Spin could consider supporting a trigger config to do this.

0 commit comments

Comments
 (0)