Skip to content

Commit 15211c5

Browse files
BigVanBigVan
committed
Add Docker runtime support for overlaybd snapshotter
Docker creates an extra init layer (key ends with -init) between image layers and container layer. This commit adds support for Docker runtime by handling the init and container layers properly. Key changes: - Add 'runtimeType' config option (containerd or docker) - Create docker.go with Docker layer detection and handling. - Update Mounts function to return correct mount options for container layer - Only rwMode 'overlayfs' is supported. For Docker accelerated images: - Init layer: attach overlaybd device, return bind mount to init/fs - Container layer: return overlay mount with lowerdir=init/fs:overlaybd_mountpoint - Remove: destroy overlaybd device when init layer is removed Signed-off-by: Yifan Yuan <tuji.yyf@alibaba-inc.com>
1 parent f8c14ca commit 15211c5

File tree

6 files changed

+583
-2
lines changed

6 files changed

+583
-2
lines changed

‎README.md‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -98,6 +98,8 @@ docker run overlaybd-convertor -r registry.hub.docker.com/library/redis -i 6.2.1
9898

9999
* See how to use TurboOCIv1 at [TurboOCIv1](docs/TURBO_OCI.md).
100100

101+
* See how to use Docker runtime with overlaybd at [DOCKER](docs/DOCKER.md).
102+
101103
* Welcome to contribute! [CONTRIBUTING](docs/CONTRIBUTING.md)
102104

103105
## Release Version Support

‎docs/DOCKER.md‎

Lines changed: 239 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,239 @@
1+
# Docker Runtime Support
2+
3+
This document describes how to use overlaybd-snapshotter with Docker runtime.
4+
5+
## Overview
6+
7+
Overlaybd-snapshotter now supports Docker runtime through containerd integration. Docker differs from containerd in that it creates an extra **init layer** (identified by `-init` suffix in the key) between image layers and the container layer.
8+
9+
When `runtimeType` is set to `"docker"`, the snapshotter handles this init layer properly:
10+
- **Init layer**: Attaches the overlaybd device and returns a bind mount
11+
- **Container layer**: Returns an overlay mount combining the init layer and overlaybd mountpoint
12+
13+
## Prerequisites
14+
15+
- Docker 20.10+ with containerd image store integration
16+
- overlaybd-snapshotter v0.6.0+
17+
- overlaybd-tcmu service running
18+
19+
## Configuration
20+
21+
### 1. Configure overlaybd-snapshotter
22+
23+
Edit `/etc/overlaybd-snapshotter/config.json`:
24+
25+
```json
26+
{
27+
"root": "/var/lib/containerd/io.containerd.snapshotter.v1.overlaybd",
28+
"address": "/run/overlaybd-snapshotter/overlaybd.sock",
29+
"runtimeType": "docker",
30+
"rwMode": "overlayfs",
31+
"verbose": "info",
32+
"logReportCaller": false,
33+
"autoRemoveDev": true,
34+
"mirrorRegistry": []
35+
}
36+
```
37+
38+
**Important:**
39+
- `runtimeType` must be set to `"docker"` (default is `"containerd"`)
40+
- `rwMode` must be `"overlayfs"` for Docker support
41+
42+
### 2. Configure Docker Daemon
43+
44+
Enable containerd snapshotter support in Docker:
45+
46+
Edit `/etc/docker/daemon.json`:
47+
48+
```json
49+
{
50+
"features": {
51+
"containerd-snapshotter": true
52+
}
53+
}
54+
```
55+
56+
Then restart Docker:
57+
58+
```bash
59+
sudo systemctl restart docker
60+
```
61+
62+
### 3. Configure containerd
63+
64+
Ensure containerd is configured to use the overlaybd snapshotter as a proxy plugin:
65+
66+
Edit `/etc/containerd/config.toml`:
67+
68+
```toml
69+
[proxy_plugins.overlaybd]
70+
type = "snapshot"
71+
address = "/run/overlaybd-snapshotter/overlaybd.sock"
72+
```
73+
74+
Restart containerd:
75+
76+
```bash
77+
sudo systemctl restart containerd
78+
```
79+
80+
## Usage
81+
82+
### Running Containers
83+
84+
Once configured, you can run overlaybd images with Docker:
85+
86+
```bash
87+
# Pull an overlaybd image
88+
docker pull registry.hub.docker.com/overlaybd/redis:7.2.3_obd
89+
90+
# Run with overlaybd snapshotter
91+
docker run --rm -it --snapshotter=overlaybd registry.hub.docker.com/overlaybd/redis:7.2.3_obd
92+
```
93+
94+
Or set overlaybd as the default snapshotter:
95+
96+
```bash
97+
# In daemon.json
98+
{
99+
"features": {
100+
"containerd-snapshotter": true
101+
},
102+
"snapshotter": "overlaybd"
103+
}
104+
```
105+
106+
### Verifying Device Creation
107+
108+
After starting a container, verify the overlaybd device:
109+
110+
```bash
111+
# Check block devices
112+
lsblk
113+
114+
# Expected output:
115+
# NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
116+
# sda 8:0 0 256G 1 disk /var/lib/containerd/io.containerd.snapshotter.v1.overlaybd/snapshots/xx/block/mountpoint
117+
118+
# Check mounts
119+
mount | grep overlaybd
120+
```
121+
122+
## How It Works
123+
124+
### Container Startup Flow
125+
126+
1. **Image Pull**: Docker pulls overlaybd format layers through containerd
127+
2. **Init Layer Preparation**:
128+
- Docker creates an init layer (key ends with `-init`)
129+
- Snapshotter attaches overlaybd device at parent's mountpoint
130+
- Returns bind mount to init layer's `fs` directory
131+
3. **Container Layer Preparation**:
132+
- Parent is the init layer
133+
- Returns overlay mount with `lowerdir=init/fs:overlaybd_mountpoint`
134+
4. **Container Startup**: Docker starts container with the prepared rootfs
135+
136+
### Device Lifecycle
137+
138+
- **Creation**: Device is created during init layer preparation
139+
- **Sharing**: Multiple containers from the same image share the same overlaybd device
140+
- **Cleanup**: Device is destroyed when the init layer is removed (after all containers exit)
141+
142+
## Limitations
143+
144+
### 1. Read-Write Mode Restrictions
145+
146+
Docker support **only works with `rwMode: "overlayfs"`**. Other modes (`dir`, `dev`) are not supported because:
147+
- Docker requires overlayfs for its layer management
148+
- Init layer mechanism is designed for overlayfs-based storage
149+
150+
### 2. Init Layer Overhead
151+
152+
Docker creates an extra init layer that:
153+
- Adds one additional layer to the overlay stack
154+
- Consumes minimal storage (usually contains only Docker metadata)
155+
- May add slight latency during container preparation
156+
157+
### 3. Device Cleanup Timing
158+
159+
Unlike containerd where devices are cleaned up immediately after container exit:
160+
- Docker may delay init layer removal
161+
- Device remains attached until init layer is garbage collected
162+
- This is normal Docker behavior and doesn't affect functionality
163+
164+
### 4. Storage Compatibility
165+
166+
- Works with **overlaybd format images** (converted or turboOCI)
167+
- Works with **OCI images** (converted on-the-fly, slower startup)
168+
- Does not support:
169+
- Native OCI images with ZFile (use turboOCI instead)
170+
- Direct block device mode (`rwMode: "dev"`)
171+
172+
### 5. Performance Considerations
173+
174+
- First container startup from an image: Full overlaybd device initialization
175+
- Subsequent containers from same image: Reuses existing device (fast)
176+
- Image pull performance: Same as containerd mode
177+
178+
### 6. Known Issues
179+
180+
- **Device busy errors**: May occur if trying to remove an image while containers are still using it. This is handled gracefully by the snapshotter.
181+
- **Init layer identification**: Relies on `-init` suffix in snapshot keys. Unusual Docker configurations may not be detected correctly.
182+
183+
## Troubleshooting
184+
185+
### Container fails to start
186+
187+
1. Check snapshotter logs:
188+
```bash
189+
sudo journalctl -u overlaybd-snapshotter -f
190+
```
191+
192+
2. Verify runtimeType configuration:
193+
```bash
194+
cat /etc/overlaybd-snapshotter/config.json | grep runtimeType
195+
```
196+
197+
3. Ensure overlaybd-tcmu is running:
198+
```bash
199+
sudo systemctl status overlaybd-tcmu
200+
```
201+
202+
### Device not cleaned up
203+
204+
Check if containers/init layers are still holding references:
205+
```bash
206+
# List active snapshots
207+
sudo ctr snapshot --snapshotter=overlaybd ls
208+
209+
# Check mounts
210+
mount | grep overlaybd
211+
```
212+
213+
### Performance issues
214+
215+
1. Enable trace recording for frequently used images
216+
2. Check network connectivity to registry
217+
3. Verify overlaybd cache settings
218+
219+
## Migration from Containerd
220+
221+
If migrating existing setups from containerd to Docker:
222+
223+
1. Images remain compatible (same overlaybd format)
224+
2. Existing pulled images can be reused
225+
3. Configuration changes required:
226+
- Add `runtimeType: "docker"` to snapshotter config
227+
- Enable containerd snapshotter in Docker daemon
228+
4. Restart services in order:
229+
- overlaybd-tcmu
230+
- overlaybd-snapshotter
231+
- containerd
232+
- Docker
233+
234+
## References
235+
236+
- [QUICKSTART](QUICKSTART.md) - General overlaybd setup
237+
- [EXAMPLES](EXAMPLES.md) - Running overlaybd containers
238+
- [TurboOCI](TURBO_OCI.md) - On-the-fly OCI image acceleration
239+
- [Configuration](../script/config.json) - Full configuration options

‎docs/QUICKSTART.md‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -85,6 +85,7 @@ The config file is `/etc/overlaybd-snapshotter/config.json`. Please create the f
8585
| ----- | ----------- |
8686
| `root` | the root directory to store snapshots. **Suggestion: This path should be a subpath of containerd's root** |
8787
| `address` | the socket address used to connect withcontainerd. |
88+
| `runtimeType` | runtime type, `"containerd"` (default) or `"docker"`. See [Docker Runtime Support](./DOCKER.md) for detail. |
8889
| `verbose` | log level, `info` or `debug` |
8990
| `rwMode` | rootfs mode about wether to use native writable layer. See [Native Support for Writable](./WRITABLE.md) for detail. |
9091
| `logReportCaller` | enable/disable the calling method |
@@ -221,6 +222,14 @@ There are several methods.
221222
sudo ctr run --net-host --snapshotter=overlaybd --rm -t registry.hub.docker.com/overlaybd/redis:6.2.1_obd demo
222223
```
223224

225+
- use Docker
226+
227+
See [Docker Runtime Support](https://github.com/containerd/accelerated-container-image/blob/main/docs/DOCKER.md) for details about using Docker with overlaybd.
228+
229+
```bash
230+
sudo docker run --net host -it --rm --snapshotter=overlaybd registry.hub.docker.com/overlaybd/redis:7.2.3_obd
231+
```
232+
224233
- use k8s/cri
225234

226235
Run with k8s or crictl, refer to [EXAMPLES_CRI](https://github.com/containerd/accelerated-container-image/blob/main/docs/EXAMPLES_CRI.md).

0 commit comments

Comments
 (0)