Skip to content

Commit 90c0c2a

Browse files
zmitchellmkenigsysndrdevusbgarbas
authored
K8s docs (#373)
Co-authored-by: Matthew Kenigsberg <[email protected]> Co-authored-by: Yannik Sander <[email protected]> Co-authored-by: Morgan Helton <[email protected]> Co-authored-by: Matthew Kenigsberg <[email protected]> Co-authored-by: Rok Garbas <[email protected]> Co-authored-by: Dan Carley <[email protected]> Co-authored-by: Daniel Sauble <[email protected]>
1 parent 28ddd82 commit 90c0c2a

File tree

14 files changed

+1104
-0
lines changed

14 files changed

+1104
-0
lines changed

‎.github/workflows/ci.yml‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -6,6 +6,7 @@ on:
66
branches:
77
- "main"
88
- "preview"
9+
- "k8s-docs"
910
pull_request:
1011

1112
permissions:

‎docs/img/containerd.png‎

121 KB
Loading

‎docs/k8s/config.md‎

Lines changed: 205 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,205 @@
1+
---
2+
title: "Configuration"
3+
description: "Configuring Imageless Kubernetes"
4+
---
5+
6+
# Configuration
7+
8+
## Authentication
9+
10+
Imageless Kubernetes allows you to run Flox environments in place of or on top of container images.
11+
Flox environments are accessed centrally via [FloxHub][floxhub] and managed using the Flox CLI.
12+
13+
In the [introduction][intro] we discussed how annotations are used to tell Imageless Kubernetes which Flox environment to run.
14+
However, we assumed that the referenced environment was publicly available without authentication.
15+
If you plan to use private environments, you will have to authenticate Imageless Kubernetes using your FloxHub user credentials.
16+
17+
To do so, you need to first login to FloxHub using the Flox CLI using [`flox auth login`][flox_auth], if you haven't done so already.
18+
You then create a new Kubernetes secret:
19+
20+
```bash
21+
flox auth token \
22+
| kubectl create secret generic floxhub-token \
23+
--from-file=floxhub-token=/dev/stdin
24+
```
25+
26+
!!! note "Token expiry"
27+
Tokens generated with `flox auth token` are associated with your user account and will expire 1 month from when they were issued. For a more robust alternative see [Machine Access Tokens for Organizations](../concepts/organizations.md#machine-access-tokens).
28+
29+
!!! note "Flox CLI version"
30+
The user creating the token via `flox auth token` will need at least version 1.7.6 of the Flox CLI.
31+
32+
Finally, you add a secret volume to your pod specification and mount it to `/var/run/secrets/flox.dev` inside your container.
33+
34+
A sample specification is shown below:
35+
36+
```yaml
37+
apiVersion: v1
38+
kind: Pod
39+
metadata:
40+
name: flox-containerd-demo
41+
annotations:
42+
flox.dev/environment: "flox/echoip"
43+
spec:
44+
runtimeClassName: flox
45+
46+
# Required for auth: a secret volume referencing the secret created with
47+
# `$ kubectl create secret`
48+
volumes:
49+
- name: secret-volume
50+
secret:
51+
secretName: floxhub-token
52+
53+
containers:
54+
- name: empty
55+
image: flox/empty:1.0.0
56+
command: ["echoip"]
57+
58+
# Required for auth: mount the secret into a known place where Imageless Kubernetes can read it.
59+
volumeMounts:
60+
- name: secret-volume
61+
mountPath: "/var/run/secrets/flox.dev"
62+
readOnly: true
63+
```
64+
65+
## Telemetry
66+
67+
Since Imageless Kubernetes uses the Flox CLI to perform certain operations such as activating your environment, Imageless Kubernetes will report the same telemetry by default that the Flox CLI reports.
68+
This includes information such as:
69+
70+
- Which subcommands were run
71+
- Which shell was used for the activation (Bash, Zsh, etc)
72+
- Whether the environment was remote (e.g. stored on FloxHub) or not
73+
- etc
74+
75+
We also use Sentry for error reporting.
76+
This information helps us focus feature development and maintenance on the areas that deliver the most value for our users.
77+
78+
However, we understand that some users either don't want any information collected or work in an environment that does not allow this kind of information to be collected.
79+
For this reason we offer the ability to disable telemetry.
80+
81+
### Disabling telemetry
82+
83+
When using the Flox CLI directly you can set `FLOX_DISABLE_METRICS=1` in your environment.
84+
With Imageless Kubernetes, you can set an annotation on the pod specification to accomplish the same goal.
85+
86+
```yaml
87+
apiVersion: v1
88+
kind: Pod
89+
metadata:
90+
name: flox-containerd-demo
91+
annotations:
92+
flox.dev/environment: "flox/echoip"
93+
# Disable telemetry
94+
flox.dev/disable-metrics: "true"
95+
spec:
96+
runtimeClassName: flox
97+
containers:
98+
- name: empty
99+
image: flox/empty:1.0.0
100+
command: ["echoip"]
101+
```
102+
103+
## Activation mode
104+
105+
By default, Imageless Kubernetes pods start in `run` mode. `run` mode is intended only to run packages installed in the Flox environment, but not provide any of the installed development dependencies.
106+
107+
The `flox.dev/activate-mode` annotation can be used to configure the mode, which may be useful for applications such as running CI jobs in Flox-enabled pods.
108+
109+
See the [`options.activate.mode`](../man/manifest.toml.md#options) option in the manifest for more details on modes.
110+
111+
```yaml
112+
apiVersion: v1
113+
kind: Pod
114+
metadata:
115+
name: quotes-app
116+
annotations:
117+
flox.dev/environment: "flox/quotes-app"
118+
# Activate in dev mode
119+
flox.dev/activate-mode: "dev"
120+
```
121+
122+
## Generations
123+
124+
A specific [generation][generations] of an environment on FloxHub can be specified as part of the `flox.dev/environment` annotation.
125+
This can be useful to pin a specific version of an environment to allow for intentional or staged upgrades.
126+
127+
```yaml
128+
apiVersion: v1
129+
kind: Pod
130+
metadata:
131+
name: quotes-app
132+
annotations:
133+
# Pin to generation 2 of the environment
134+
flox.dev/environment: "flox/quotes-app:2"
135+
```
136+
137+
## Mutability
138+
139+
By default, Imageless Kubernetes pods are immutable, such that `flox install` commands are not possible and `/nix` is mounted read-only.
140+
141+
To enable mutability (e.g. for debugging), the `flox.dev/nix-mutable` annotation can be used.
142+
143+
```yaml
144+
apiVersion: v1
145+
kind: Pod
146+
metadata:
147+
name: quotes-app
148+
annotations:
149+
flox.dev/environment: "flox/quotes-app"
150+
# Enable /nix mutability
151+
flox.dev/nix-mutable: "true"
152+
```
153+
154+
## Mixed Flox/non-Flox pods
155+
156+
Imageless Kubernetes allows mixing Flox and non-Flox-based containers in the same pod, supporting the use of conventional init or sidecar containers combined with Flox-based workloads.
157+
This is accomplished through the use of two annotations: `flox.dev/skip-containers` and `flox.dev/skip-containers-exec`.
158+
159+
`flox.dev/skip-containers` accepts a comma-separated list of containers that will _not_ be modified by the Flox runtime, and will be run as if they were started with the default runtime (e.g. `runc`). This option is best used for sidecars like `vault-agent` or `istio` that should run completely unmodified.
160+
161+
```yaml
162+
apiVersion: v1
163+
kind: Pod
164+
metadata:
165+
name: quotes-app
166+
annotations:
167+
vault.hashicorp.com/agent-inject: "true"
168+
vault.hashicorp.com/role: "myapp-role"
169+
flox.dev/environment: "flox/quotes-app"
170+
# Keep these containers unmodified
171+
flox.dev/skip-containers: "vault-agent,vault-agent-init"
172+
spec:
173+
containers:
174+
...
175+
- name: vault-agent
176+
image: hashicorp/vault:latest
177+
command: ["/bin/sh", "-ec"]
178+
args:
179+
- |
180+
vault agent -config=/vault/configs/agent.hcl
181+
env:
182+
- name: VAULT_ADDR
183+
value: "http://vault.vault.svc.cluster.local:8200"
184+
volumeMounts:
185+
- name: vault-secrets
186+
mountPath: /vault/secrets
187+
188+
- name: quotes-app
189+
image: flox/empty:1.0.0
190+
command: ["quotes-app-go"]
191+
volumeMounts:
192+
- name: vault-secrets
193+
mountPath: /vault/secrets
194+
readOnly: true
195+
...
196+
```
197+
198+
`flox.dev/skip-containers-exec` also accepts a comma separated list of containers, but containers specified in this annotation _will_ contain the Flox environment specified in `flox.dev/environment`.
199+
200+
The difference from `skip-containers` is that while `skip-containers-exec` containers will have their main process run from the Flox environment, commands run via `kubectl exec` or equivalent will be run outside of it. This option is best used when you want the container's main workload to run in the Flox environment, but need exec commands (for debugging, health checks, or auxiliary tasks) to run in the base container environment without Flox.
201+
202+
[intro]: ./intro.md
203+
[floxhub]: ../concepts/floxhub.md
204+
[flox_auth]: ../man/flox-auth.md
205+
[generations]: ../concepts/generations.md

‎docs/k8s/examples/gitlab-ci.md‎

Lines changed: 69 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,69 @@
1+
---
2+
title: "GitLab CI"
3+
description: "Demo running GitLab CI with Imageless Kubernetes"
4+
---
5+
6+
You can use Imageless Kubernetes in many applications where you would rely on a conventional image-based workflow.
7+
8+
This example shows how you can use Imageless Kubernetes with GitLab CI, running jobs inside of the same Flox environment you use for development.
9+
10+
## GitLab runner configuration
11+
12+
To configure your GitLab runner to run Imageless Kubernetes pods, add this section to the runner's `config.toml`:
13+
14+
```toml
15+
[[runners]]
16+
[runners.kubernetes]
17+
namespace = {% raw %}"{{ default .Release.Namespace .Values.runners.jobNamespace }}"{% endraw %}
18+
runtime_class_name = "flox"
19+
image = "flox/empty:1.0.0"
20+
pod_annotations_overwrite_allowed = '^flox\.dev.*'
21+
[runners.kubernetes.pod_annotations]
22+
"flox.dev/skip-containers" = "init-permissions,helper"
23+
"flox.dev/skip-containers-exec" = "build"
24+
"flox.dev/activate-mode" = "dev" # optional
25+
```
26+
27+
!!! note "Note"
28+
These options can also be passed as part of the job definition in `.gitlab-ci.yml`, see the [GitLab documentation][gitlab-k8s-docs] for more details.
29+
30+
These settings will start all GitLab job pods using the Flox runtime, with an empty container image, and allow setting additional annotations (e.g. `flox.dev/environment`) on a per-job basis.
31+
32+
The `flox.dev/skip-containers` and `flox.dev/skip-containers-exec` annotations are necessary to allow GitLab's init containers to get the code and job script into build container for execution.
33+
34+
`flox.dev/activate-mode` is set to make build dependencies available to the job script.
35+
36+
See the [configuration][config] page for more details on annotations.
37+
38+
## GitLab job configuration
39+
40+
Once you've configured the runner, you will need to supply each job with the desired `flox.dev/environment` annotation, which can be done in `.gitlab-ci.yml`:
41+
42+
```yaml
43+
stages:
44+
- hello
45+
46+
hello-job:
47+
stage: hello
48+
variables:
49+
KUBERNETES_POD_ANNOTATIONS_1: "flox.dev/environment=flox/hello"
50+
script:
51+
- hello
52+
```
53+
54+
where the value of the annotation is the name of an environment from [FloxHub][floxhub].
55+
56+
!!! note "Note"
57+
The `flox.dev/environment` annotation is *not* optional -- pods will fail to start if it is not supplied.
58+
59+
## Conclusion
60+
61+
Now, any job you target to this runner will be executed with Imageless Kubernetes.
62+
63+
If you utilize the same Flox environment used for development, you will be able to seamlessly test with the exact same packages, regardless of what system or architecture is used.
64+
65+
Since the CI environment doesn’t rely on a container image, updates are instant: run `flox push`, and the job will pick up the changes automatically.
66+
67+
[gitlab-k8s-docs]: https://docs.gitlab.com/runner/executors/kubernetes/
68+
[config]: ../config.md
69+
[floxhub]: ../../concepts/floxhub.md

‎docs/k8s/examples/kind-demo.md‎

Lines changed: 115 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,115 @@
1+
---
2+
title: "Web server with Redis"
3+
description: "Demo running a simple web server backed by Redis on kind"
4+
---
5+
6+
With Imageless Kubernetes, you can run a container just like in any other
7+
Kubernetes deployment, but you don't have to build a container image.
8+
9+
To demonstrate this, we'll run a simple web server backed by Redis using two
10+
Flox environments.
11+
12+
## Running the example
13+
14+
The entirety of this example can be run locally with the following commands:
15+
16+
```bash
17+
git clone https://github.com/flox/flox-kind-demo.git
18+
cd flox-kind-demo
19+
flox activate
20+
just up
21+
```
22+
23+
This starts a local Kubernetes cluster using `kind` and deploys a web server
24+
backed by Redis.
25+
26+
To fetch something from the web server, run:
27+
28+
```bash
29+
curl localhost:3000/quotes/0
30+
```
31+
32+
## Quotes app environment
33+
34+
The example runs a deployment of [`flox/quotes-app`](https://hub.flox.dev/flox/quotes-app), which is just like any other
35+
Kubernetes deployment, but with a few key differences.
36+
Here's a snippet from the deployment manifest:
37+
38+
```yaml
39+
metadata:
40+
labels:
41+
app: quotes
42+
annotations:
43+
flox.dev/environment: "flox/quotes-app"
44+
spec:
45+
runtimeClassName: flox
46+
containers:
47+
- name: quotes
48+
image: flox/empty:1.0.0
49+
command: ["quotes-app-go", "-r", "redis:6379"]
50+
```
51+
52+
The full deployment manifest can be found in the
53+
[flox-kind-demo repo](https://github.com/flox/flox-kind-demo/blob/main/quotes.yaml).
54+
55+
There are two key lines needed to run the container using a Flox environment
56+
instead of a container image:
57+
58+
- Specifying `runtimeClassName: flox` runs the container using the Flox
59+
containerd shim.
60+
- The annotation `flox.dev/environment: "flox/quotes-app"` specifies the Flox
61+
environment to use to bootstrap the container filesystem instead of a container image
62+
63+
Just as with any container, you can specify a startup command, which is
64+
`["quotes-app-go", "-r", "redis:6379"]` in this case.
65+
66+
When the container starts, the [`flox/quotes-app`](https://hub.flox.dev/flox/quotes-app) Flox environment is pulled from FloxHub and bind mounted into the container.
67+
This environment contains the [`flox/quotes-app-go`](https://hub.flox.dev/packages/flox/quotes-app-go) package, which is a simple web server published to FloxHub.
68+
When the container starts, the environment is activated, and then `quotes-app-go` is run inside the activated environment.
69+
70+
### Redis environment
71+
72+
The `quotes-app-go` server uses a Redis instance running in a second deployment.
73+
Just like the first pod, rather than specifying a container image, the Redis
74+
deployment runs the environment `flox/redis` which is pulled from
75+
[hub.flox.dev/flox/redis](https://hub.flox.dev/flox/redis).
76+
77+
Here's the relevant snippet from the Redis deployment manifest:
78+
79+
```yaml
80+
metadata:
81+
labels:
82+
app: redis
83+
annotations:
84+
flox.dev/environment: "flox/redis"
85+
spec:
86+
runtimeClassName: flox
87+
containers:
88+
- name: redis
89+
image: flox/empty:1.0.0
90+
command: ["redis-server", "--daemonize", "no", "--dir", "/data", "--bind", "0.0.0.0", "--protected-mode", "no" ]
91+
volumeMounts:
92+
- name: redis-data
93+
mountPath: /data
94+
```
95+
96+
The full deployment manifest can be found in the
97+
[flox-kind-demo repo](https://github.com/flox/flox-kind-demo/blob/main/redis.yaml).
98+
99+
## Updating the deployment
100+
101+
Because the environment is hosted on FloxHub, there's no need to rebuild a
102+
container image to update the deployment.
103+
After a change to `quotes-app-go`, updating the deployment would require running
104+
a `flox publish` for `quotes-app-go` and a `flox upgrade -r flox/quotes-app`.
105+
After that, restarting a pod will pull the latest generation of the environment.
106+
This allows deploying software with the reproducibility of a container, but
107+
without the overhead of having to rebuild an entire container image when iterating.
108+
109+
## Cleaning up
110+
111+
To tear down the local kind cluster, run:
112+
113+
```bash
114+
just down
115+
```

0 commit comments

Comments
 (0)