Skip to content

Commit 5bf2178

Browse files
Kern--Kern--
authored
Merge pull request #686 from Kern--/remote-snapshotter-docs
Add remote snapshotter getting started guide
2 parents d5b3cd2 + a1f0eed commit 5bf2178

File tree

2 files changed

+203
-0
lines changed

2 files changed

+203
-0
lines changed

‎docs/remote-snapshotter-getting-started.md

Lines changed: 172 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,172 @@
1+
# Getting started with remote snapshotters in firecracker-containerd
2+
3+
This guide will walk you through setting up firecracker-containerd to work with remote snapshotters. By the end of the guide, you should have a working firecracker-containerd setup that allows you to lazily load container images inside network-connected Firecracker microVMs.
4+
5+
We support the [stargz-snapshotter](https://github.com/containerd/stargz-snapshotter) as a remote snapshotter in firecracker-containerd. The stargz-snapshotter uses a specially formatted OCI container image in a form called estargz. The estargz format adds metadata to the image which allows the stargz-snapshotter to randomly access individual files in the image over the network rather than at the layer level. This enables the stargz-snapshotter to start containers before the image is fully downloaded and lazily load additional parts of the container at run time. For more information, please read the stargz-snapshotter docs including the doc on the [estargz image format](https://github.com/containerd/stargz-snapshotter/blob/v0.11.4/docs/estargz.md).
6+
7+
# Setup
8+
9+
## Prerequisites
10+
11+
You will need the following to use firecracker-containerd with support for remote snapshotters:
12+
13+
* A computer with a recent-enough version of Linux (4.14+), and KVM enabled
14+
15+
* A recent installation of Docker CE (for building the Firecracker root filesystem)
16+
17+
* git
18+
19+
* Go 1.17 or later
20+
21+
* This repository cloned onto your local machine
22+
23+
For more details about these requirements please see the base [getting started guide](https://github.com/firecracker-microvm/firecracker-containerd/blob/57126558036c4a34a02967865467b2161ca9227e/docs/getting-started.md)
24+
25+
26+
## Build firecracker-containerd
27+
28+
The following commands will build firecracker-containerd (including the necessary shim and agent) as well as Firecracker. It will install these binaries to `/usr/local/bin`. You do not have to install any of the components in this guide, however all of the example configuration assumes the components have been installed and will need modification if you choose to skip this step.
29+
30+
```
31+
make
32+
make install
33+
make firecracker
34+
make install-firecracker
35+
```
36+
37+
## Build a Linux kernel with FUSE support
38+
39+
We generally recommend that users download our AWS-provided Linux kernel that is known to work with Firecracker. Unfortunately this kernel does not support FUSE which is needed for remote snapshotters. We do provide make rules to build a working kernel with FUSE support:
40+
41+
```
42+
make kernel
43+
make install-kernel
44+
```
45+
46+
By default, this will build a Linux 4.14 kernel. If you would like to use 5.10, use:
47+
```
48+
KERNEL_VERSION=5.10 make kernel
49+
KERNEL_VERSION=5.10 make install-kernel
50+
```
51+
52+
[Note: Firecracker only officially supports 4.14 and 5.10](https://github.com/firecracker-microvm/firecracker/blob/v1.1.0/docs/kernel-policy.md).
53+
54+
55+
## Build a Firecracker rootfs with a remote snapshotter
56+
57+
The firecracker-containerd repository includes an image builder component that constructs a Debian-based root filesystem for the VM. The default installs the necessary firecracker-containerd agent as well as sets up an overlay to allow re-using the read only image across multiple VMs. We also supply a variant that provides these same features and additionally bundles the stargz-snapshotter.
58+
59+
```
60+
make image-stargz
61+
make install-stargz-rootfs
62+
```
63+
64+
## Setup the demo network
65+
66+
While networking support is optional for firecracker-containerd in general, it is required for remote snapshotter support because the remote snapshotter will lazily load container images from inside the VM. The easiest way to set this up is to use the provided `demo-network` which will install CNI configuration to create a bridge, a veth pair connected to the bridge, and an adapter to connect the tap to Firecracker.
67+
68+
```
69+
make demo-network
70+
```
71+
72+
## Configure Firecracker
73+
74+
If you followed the getting started or quickstart guide to setup firecracker-containerd, then Firecracker will be configured to use the default rootfs and kernel without FUSE support. In order for firecracker-containerd to work with remote snapshotters you should update `/etc/containerd/firecracker-runtime.json` to set:
75+
76+
* `kernel_image_path` to point to your kernel built with FUSE support
77+
* `root_drive` to points to your rootfs with a remote snapshotter
78+
* `default_network_interfaces` to set up your CNI network. This network interface must be allowed to access MMDS in order to configure credentials for the remote snapshotter
79+
80+
An example of this type of configuration is as follows
81+
```
82+
{
83+
"firecracker_binary_path": "/usr/local/bin/firecracker",
84+
"kernel_image_path": "/var/lib/firecracker-containerd/runtime/default-vmlinux.bin",
85+
"kernel_args": "console=ttyS0 pnp.debug=1 noapic reboot=k panic=1 pci=off nomodules ro systemd.unified_cgroup_hierarchy=0 systemd.journald.forward_to_console systemd.unit=firecracker.target init=sbin/overlay-init",
86+
"root_drive": "/var/lib/firecracker-containerd/runtime/rootfs-stargz.img",
87+
"log_fifo": "fc-logs.fifo",
88+
"log_levels": ["debug"],
89+
"metrics_fifo": "fc-metrics.fifo",
90+
"default_network_interfaces": [
91+
{
92+
"AllowMMDS": true,
93+
"CNIConfig": {
94+
"NetworkName": "fcnet",
95+
"InterfaceName": "veth0"
96+
}
97+
}
98+
]
99+
}
100+
```
101+
102+
## Configure containerd
103+
104+
Firecracker-containerd relies on a `demux-snapshotter` running on the host to proxy snapshotter requests to the appropriate remote snapshotter running in the appropriate VM. `/etc/firecracker-containerd/config.toml` needs to be updated to tell `firecracker-containerd` about this `demux-snapshotter`.
105+
106+
The following configuration will create a snapshotter called `proxy` which will be redirected over gRPC to the default endpoint for the `demux-snapshotter`
107+
108+
```
109+
[proxy_plugins]
110+
[proxy_plugins.proxy]
111+
type = "snapshot"
112+
address = "/var/lib/demux-snapshotter/snapshotter.sock"
113+
```
114+
115+
## Configure demux-snapshotter
116+
117+
To configure the demux-snapshotter, add the following config to `/etc/demux-snapshotter/config.toml`
118+
119+
```
120+
[snapshotter.listener]
121+
type = "unix"
122+
address = "/var/lib/demux-snapshotter/snapshotter.sock"
123+
124+
[snapshotter.proxy.address.resolver]
125+
type = "http"
126+
address = "http://127.0.0.1:10001"
127+
128+
[snapshotter.metrics]
129+
enable = false
130+
131+
[debug]
132+
logLevel = "debug"
133+
```
134+
135+
Create the demux snapshotter's runtime directory.
136+
```
137+
mkdir -p /var/lib/demux-snapshotter
138+
```
139+
140+
## Start all of the host-daemons
141+
142+
Remote snapshotters require 3 host-side daemons:
143+
* `firecracker-containerd` - For running VMs/containers
144+
* `demux-snapshotter` - For demultiplexing snapshotter requests from firecracker-containerd to the appropriate remote-snapshotter in the appropriate VM
145+
* `http-address-resolver` - For mapping containerd namespace to VM vsock address (see [limitations](#limitations))
146+
147+
Run each in a separate shell:
148+
149+
```
150+
firecracker-containerd --config /etc/firecracker-containerd/config.toml
151+
```
152+
```
153+
snapshotter/demux-snapshotter
154+
```
155+
```
156+
snapshotter/http-address-resolver
157+
```
158+
159+
## Pull a remote image
160+
161+
Pulling a remote image requires special configuration which is currently not implemented by any CLI tool. We do have an example client implementation which can be built or extended for testing. Please see the example here: https://github.com/firecracker-microvm/firecracker-containerd/tree/main/examples/cmd/remote-snapshotter
162+
163+
## Limitations
164+
165+
### Containerd namespace must be 1-1 with a VM
166+
Portions of containerd's snapshotter API receive only containerd namespace and snapshotter key. In order to route those requests to the correct remote snapshotter running inside a VM, we route based on namespace. This means that we must have a unique mapping between a containerd namespace and a VM in order for requests to be fulfilled.
167+
168+
### OCI Images must be converted to estargz
169+
estargz images are compatible with OCI images, but OCI images must be converted to estargz to be usable with the lazy loading properties of the stargz-snapshotter. For information on how to convert and image, please see the [stargz-snapshotter's documentation](https://github.com/containerd/stargz-snapshotter/blob/v0.11.4/docs/ctr-remote.md)
170+
171+
### There is no fallback to ahead-of-time pull
172+
The stargz-snapshotter can normally fall back to an embedded overlay snapshotter with ahead-of-time pull if the image is not estargz. This does not work with firecracker-containerd because in this scenario, the client is responsible for unpacking the container image into the mount points provided by the snapshotter, however the client runs on the host and the mount points are isolated inside the VM.

‎examples/cmd/remote-snapshotter/README.md

Lines changed: 31 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,31 @@
1+
# Remote Snapshotter Example
2+
3+
This example shows how to build a firecracker-containerd client that can launch lazily-loaded estargz images in a firecracker microvm. If you haven't already, please follow the [remote snapshotter getting started guide.](https://github.com/firecracker-microvm/firecracker-containerd/blob/main/docs/remote-snapshotter-getting-started.md)
4+
5+
# Building
6+
7+
The example can be built with
8+
9+
```
10+
make remote-snapshotter
11+
```
12+
13+
# Usage
14+
To use the example, run (e.g.)
15+
16+
```
17+
./remote-snapshotter ghcr.io/firecracker-microvm/firecracker-containerd/amazonlinux:latest-esgz
18+
```
19+
20+
A list of additional, pre-converted images for testing can be found in the [stargz-snapshotter documentation](https://github.com/containerd/stargz-snapshotter/blob/main/docs/pre-converted-images.md)
21+
22+
23+
This will launch the container inside a microVM with lazy loading supplied by the stargz-snaphotter and connect stdio to the container.
24+
25+
Note: stargz-snapshotter has a fallback mechanism to use an embedded overlay snapshotter if an image is not in estargz format. This will not work with firecracker-containerd because the client is responsible for writing content into the snapshot, but the snapshot mountpoints are isolated inside the VM.
26+
27+
# Credentials
28+
29+
By default, the example tool will ask for a docker username/password each time it is invoked. It can also read these from environment the variables `$DOCKER_USERNAME` and `$DOCKER_PASSWORD`.
30+
31+
Note that this means it does not support multiple container registries at the same time, nor credential helpers for retrieving credentials.

0 commit comments

Comments
 (0)