Skip to content

Commit 02a98ea

Browse files
sondavidbsondavidb
committed
Added snapshot documentation + pipeline example code
Signed-off-by: David Son <[email protected]>
1 parent d48665a commit 02a98ea

File tree

7 files changed

+1791
-0
lines changed

7 files changed

+1791
-0
lines changed

‎.buildkite/pipeline.yml

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -102,6 +102,15 @@ steps:
102102
queue: "${BUILDKITE_AGENT_META_DATA_QUEUE:-default}"
103103
distro: "${BUILDKITE_AGENT_META_DATA_DISTRO}"
104104
hostname: "${BUILDKITE_AGENT_META_DATA_HOSTNAME}"
105+
106+
- label: ':book: examples'
107+
commands:
108+
- "sudo -E PATH=$PATH FC_TEST_DATA_PATH=${FC_TEST_DATA_PATH} make -C examples/cmd/snapshotting run"
109+
- "sudo -E PATH=$PATH FC_TEST_DATA_PATH=${FC_TEST_DATA_PATH} make -C examples/cmd/snapshotting clean"
110+
agents:
111+
queue: "${BUILDKITE_AGENT_META_DATA_QUEUE:-default}"
112+
distro: "${BUILDKITE_AGENT_META_DATA_DISTRO}"
113+
hostname: "${BUILDKITE_AGENT_META_DATA_HOSTNAME}"
105114

106115
- label: ':hammer: tests'
107116
commands:

‎docs/snapshotting.md

Lines changed: 58 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,58 @@
1+
# Snapshotting
2+
3+
Snapshotting is currently supported in the Firecracker Go SDK using Firecracker v1.0.0's API.
4+
5+
Due to [known issues and limitations](https://github.com/firecracker-microvm/firecracker/blob/firecracker-v1.0/docs/snapshotting/snapshot-support.md#known-issues-and-limitations), it is currently not recommended to use snapshots in production.
6+
7+
Snapshots created in this version only save the following:
8+
- guest memory
9+
- emulated hardware state (both KVM & Firecracker emulated hardware)
10+
11+
Each of the above are saved in its own separate file. Anything else is up to the user to restore (tap devices, drives, etc.).
12+
13+
In particular, drives must be in the same location as they were when loading the snapshot. Otherwise, the API call will fail. Changing said drive file can lead to some unexpected behaviors, so it is recommended to make minimal changes to the drive.
14+
15+
Snapshots can only be loaded upon device startup. Upon loading the snapshot, the emulated hardware state is restored, and normal VM activites can resume right where they left off.
16+
17+
Read more in-depth documentation on Firecracker's snapshotting tool [here](https://github.com/firecracker-microvm/firecracker/blob/firecracker-v1.0/docs/snapshotting/snapshot-support.md).
18+
19+
## Using Snapshots via Firecracker Go SDK
20+
21+
Snapshots can be created via a machine object's `CreateSnapshot()` function. The call will make the snapshot files at the specified paths, with the memory saved to `memPath`, and the machine state saved to `snapPath`. The VM must be paused beforehand.
22+
23+
```
24+
import (
25+
sdk "github.com/firecracker-microvm/firecracker-go-sdk"
26+
)
27+
28+
...
29+
30+
ctx := context.Background()
31+
cfg := sdk.Config{
32+
33+
...
34+
35+
}
36+
37+
m, _ := sdk.NewMachine(ctx, cfg)
38+
m.Start(ctx)
39+
m.PauseVM(ctx)
40+
m.CreateSnapshot(ctx, memPath, snapPath)
41+
```
42+
43+
The snapshot can be loaded at any later time at startup of a machine via the machine's `Start()` function, using `WithSnapshot()` as an option. The VM must then be resumed before attempting to use it.
44+
45+
```
46+
ctx := context.Background()
47+
cfg := sdk.Config{
48+
49+
...
50+
51+
}
52+
m, _ := sdk.NewMachine(ctx, cfg)
53+
54+
m.Start(ctx, sdk.WithSnapshot(memPath, snapPath))
55+
m.ResumeVM(ctx)
56+
```
57+
58+
Check out [examples/cmd/snapshotting](../examples/cmd/snapshotting) for a quick example that can be run on your machine.

‎examples/cmd/snapshotting/Makefile

Lines changed: 73 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,73 @@
1+
# Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved.
2+
#
3+
# Licensed under the Apache License, Version 2.0 (the "License"). You may
4+
# not use this file except in compliance with the License. A copy of the
5+
# License is located at
6+
#
7+
# http://aws.amazon.com/apache2.0/
8+
#
9+
# or in the "license" file accompanying this file. This file is distributed
10+
# on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
11+
# express or implied. See the License for the specific language governing
12+
# permissions and limitations under the License.
13+
14+
RELEASE_URL=https://github.com/firecracker-microvm/firecracker/releases
15+
VER=v1.0.0
16+
17+
ARCH=$(shell uname -m)
18+
GID = $(shell id -g)
19+
20+
PWD=$(shell pwd)
21+
FC_TEST_DATA_PATH?=$(PWD)
22+
23+
all: plugins image vmlinux firecracker
24+
25+
plugins: bin/tc-redirect-tap bin/ptp bin/host-local | bin
26+
27+
bin:
28+
mkdir -p bin
29+
30+
bin/tc-redirect-tap: bin
31+
GO111MODULE=off GOBIN=$(PWD)/bin \
32+
go get github.com/awslabs/tc-redirect-tap/cmd/tc-redirect-tap
33+
34+
bin/ptp: bin
35+
GO111MODULE=off GOBIN=$(PWD)/bin \
36+
go get github.com/containernetworking/plugins/plugins/main/ptp
37+
38+
bin/host-local: bin
39+
GO111MODULE=off GOBIN=$(PWD)/bin \
40+
go get github.com/containernetworking/plugins/plugins/ipam/host-local
41+
42+
image:
43+
ifeq ($(GID), 0)
44+
- cp ${FC_TEST_DATA_PATH}/root-drive-with-ssh.img root-drive-with-ssh.img
45+
- cp ${FC_TEST_DATA_PATH}/root-drive-ssh-key root-drive-ssh-key
46+
$(MAKE) root-drive-with-ssh.img root-drive-ssh-key
47+
else
48+
$(error unable to place ssh key without root permissions)
49+
endif
50+
51+
vmlinux:
52+
curl --location -o vmlinux https://s3.amazonaws.com/spec.ccfc.min/img/quickstart_guide/${ARCH}/kernels/vmlinux.bin
53+
54+
firecracker:
55+
curl -L ${RELEASE_URL}/download/${VER}/firecracker-${VER}-${ARCH}.tgz | tar -xz
56+
mv release-${VER}-${ARCH}/firecracker-${VER}-${ARCH} firecracker
57+
rm -rf release-${VER}-${ARCH}
58+
59+
root-drive-with-ssh.img root-drive-ssh-key:
60+
- mkdir temp
61+
- git clone https://github.com/firecracker-microvm/firecracker temp
62+
temp/tools/devtool build_rootfs
63+
cp temp/build/rootfs/bionic.rootfs.ext4 root-drive-with-ssh.img
64+
cp temp/build/rootfs/ssh/id_rsa root-drive-ssh-key
65+
rm -rf temp
66+
67+
run: all
68+
go run example_demo.go
69+
70+
clean:
71+
rm -rf bin firecracker root-drive-ssh-key root-drive-with-ssh.img vmlinux
72+
73+
.PHONY: all clean image plugins run

‎examples/cmd/snapshotting/README.md

Lines changed: 81 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,81 @@
1+
# Snapshotting demo
2+
3+
This example shows snapshotting in action by sending a marker to a VM via a running process (in this case `sleep 422`), snapshotting the VM, closing it, loading and starting a new machine via the same snapshot, and checking for the marker.
4+
5+
This test requires both KVM and root access.
6+
7+
## Running the test
8+
9+
Run this test by first running
10+
11+
```
12+
sudo -E env PATH=$PATH make all
13+
```
14+
15+
followed by
16+
17+
```
18+
sudo -E env PATH=$PATH go run example_demo.go
19+
```
20+
21+
Alternatively, to do both of the above,
22+
```
23+
sudo -E env PATH=$PATH make run
24+
```
25+
26+
Note the user PATH variable is different from the root user's PATH variable, hence the need for `-E env PATH=$PATH`.
27+
28+
Upon running, the VM logs will be printed to the console, as well as the IP of the VM. It will then show that it is sending the marker (in our case, `sleep 422`).
29+
30+
Afterwards, the snapshot is created and the machine is terminated. The snapshot files are saved in the snapshotssh folder created in the directory.
31+
32+
Then, a new machine is created, booted with the snapshot that was just taken, and the IP of the VM will once again be printed to the console (which should be the same as the last machine). The output of searching for the marker (in our case `ps -aux | grep "sleep 422"`) is then printed to the console and the user can confirm that the snapshot loaded properly.
33+
34+
To run this test more dynamically, you can pause the execution of the program after starting the machine (i.e. after the call to m.Start() and the IP is shown on the screen).
35+
36+
```
37+
err = m.Start()
38+
39+
...
40+
41+
vmIP := m.Cfg.NetworkInterfaces[0].StaticConfiguration.IPConfiguration.IPAddr.IP.String()
42+
fmt.Printf("IP of VM: %v\n", vmIP)
43+
fmt.Scanln() // block, allows you to ssh from another shell
44+
45+
...
46+
47+
err = m.Start()
48+
49+
...
50+
51+
fmt.Println("Snapshot loaded")
52+
fmt.Printf("IP of VM: %v\n", ipToRestore)
53+
fmt.Scanln() // block, allows you to ssh from another shell
54+
```
55+
56+
```
57+
sudo ssh -i root-drive-ssh-key root@[ip]
58+
```
59+
60+
Pressing enter resumes execution of the program.
61+
62+
You can remove dependencies via a simple `make clean`.
63+
64+
```
65+
sudo make clean
66+
```
67+
68+
## Issues
69+
70+
You may encounter an issue where the image does not build properly. This is often indicated via the following near the end of terminal output:
71+
72+
```
73+
umount: /firecracker/build/rootfs/mnt: not mounted.
74+
```
75+
76+
This is due to an issue in Firecracker's devtool command used to dynamically create an image. Fixing this is often as simple as rerunning the command.
77+
78+
```
79+
sudo rm -rf root-drive-with-ssh.img root-drive-ssh-key
80+
sudo make image
81+
```

0 commit comments

Comments
 (0)