Skip to content

Commit 55fb778

Browse files
dallasd1dallasd1
committed
docs: add dm-verity image layer signing (notaryproject#1335)
This proposal discusses adding per-layer container image signing using the PKCS#7 format. This will enable signing individual container image layers that are later verified by the kernel at runtime. Runtime verification also depends on milestone 1 of this [RFC for code integrity](containerd/containerd#12081) in containerd. At the time of writing, milestone 1.2 is in PR review and milestone 1.3 remains. --------- Signed-off-by: Dallas Delaney <dadelan@microsoft.com>
1 parent 4952c1c commit 55fb778

File tree

1 file changed

+202
-0
lines changed

1 file changed

+202
-0
lines changed

‎specs/proposals/dm-verity.md‎

Lines changed: 202 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,202 @@
1+
# Proposal: Code Integrity for OCI Containers – Per-Layer DM-Verity Signing with Notation
2+
3+
## Overview
4+
5+
This proposal extends the Notation CLI to support signing and verifying OCI container image layers with dm-verity Merkle tree root hashes. While Notation currently supports signing container images at the manifest level, this proposal enables container image per-layer integrity protection that can be continuously enforced at runtime using dm-verity and [Integrity Policy Enforcement](https://docs.kernel.org/next/admin-guide/LSM/ipe.html) (IPE). Container layer DM-verity signing is critical for extending the trust from kernel code-integrity guarantees to workloads within trusted containers, while blocking the execution of untrusted containers and binaries created in mutable container state.
6+
7+
## Problem Statement & Motivation
8+
9+
Modern Linux container hosts may achieve a high level of security by running an immutable host OS, preventing tampering with system binaries. However, OCI container images themselves have traditionally not been held to the same standard – integrity is only verified at image pull time, with no continuous enforcement at runtime. This leaves a gap where if an attacker injects or executes a malicious binary inside a container, the host has no built-in mechanism to prevent it from running.
10+
11+
In developing [Azure Linux with OS Guard](https://techcommunity.microsoft.com/blog/linuxandopensourceblog/azure-linux-with-os-guard-immutable-container-host-with-code-integrity-and-open-/4437473) we aim to to extend code integrity protections into OCI containers using dm-verity and IPE. Each container image layer is backed by a read-only dm-verity block device whose integrity is ensured by a Merkle tree root hash. The root hash is signed by a key that the Linux kernel trusts. At container start, the kernel verifies each root hash signature before allowing the layer to mount. IPE policies then allow execution only from layers with verified hashes.
12+
13+
Signing container images at the manifest level alone is not sufficient to ensure continuous runtime integrity. While manifest signatures protect against image tampering during distribution (image pull), they do not enable enforcement at runtime. Container layers must also have kernel-verifiable signatures to ensure their integrity. With the increasing adoption of immutable infrastructure and zero-trust security models, securing these artifacts with continuous kernel enforcement is critical. By adding per-layer container image signing, Notation can extend its capabilities to enable kernel-enforced container integrity.
14+
15+
The challenge is that OCI registries and container image tools currently support distributing signatures but there is no good tooling for creating per-layer signatures. Existing Notation signatures use JWS or COSE formats, which cannot be verified by the Linux kernel. Kernel-level dm-verity enforcement requires each layer's dm-verity root hash to be accompanied by a PKCS#7 signature that the kernel can check at mount time.
16+
17+
## Scenarios
18+
19+
The following describes how per-layer dm-verity signing can enhance container security across different attack scenarios:
20+
21+
### Scenario 1: Runtime Layer Tampering (Current Implementation)
22+
23+
Sarah, a DevOps engineer, deploys a containerized application to a production Kubernetes cluster. The container images are signed using standard Notation manifest signatures. An attacker gains access to a worker node and modifies one of the container layers on the host filesystem by injecting a malicious binary. When the container restarts, the modified layer is mounted without detection because:
24+
- Notation's manifest signature only verifies the image at pull time
25+
- The digest in the manifest still matches the original layer blob in the registry
26+
- The layer is mounted without verification of root hashes after being tampered with offline (offline attack allowed)
27+
- The malicious binary cannot be detected at runtime since IPE is not present to prevent binaries executing from unsigned dm-verity volumes (runtime attack allowed)
28+
29+
30+
The malicious binary executes successfully, compromising the application. Current Notation signatures cannot prevent this attack because they don't provide continuous runtime enforcement at the image layer level.
31+
32+
### Scenario 2: Kernel-Enforced Layer Integrity (Proposed Solution)
33+
34+
With the proposed per-layer dm-verity signing:
35+
36+
David, a platform engineer, uses Notation with the proposed dm-verity signing changes. The following happens:
37+
1. Each container image layer is processed to generate an EROFS filesystem with dm-verity metadata
38+
2. The dm-verity Merkle tree root hash for each layer is computed deterministically
39+
3. Each root hash is signed using PKCS#7 format with the company's signing key
40+
4. Image layer signatures and metadata are injected into a referrer artifact attached to the image manifest in the registry
41+
42+
When containers are deployed:
43+
1. The EROFS containerd snapshotter fetches the OCI image and its attached referrer artifact that contains the layer signatures from the registry
44+
2. For each layer, the snapshotter creates a dm-verity block device, passing the root hash and PKCS#7 signature to the kernel
45+
3. The Linux kernel verifies the PKCS#7 signed root hash against trusted keys before mounting the layer (offline attack blocked)
46+
4. IPE policies enforce that only correctly signed dm-verity volumes can execute code at runtime (runtime attack blocked)
47+
48+
If an attacker attempts to modify a layer like in the first scenario, the root hash verification fails immediately and the kernel refuses to mount the tampered layer. If an attacker drops an unsigned binary into a running container and tries to execute it, IPE blocks execution because it was not loaded from a signed dm-verity volume.
49+
50+
## Proposal
51+
52+
This section outlines the proposed solution for signing and verifying OCI container image layers with dm-verity root hashes using Notation CLI commands. The following topics are outside the scope of this document:
53+
54+
- Detailed command usage, which will be covered in individual command specifications
55+
- [EROFS](https://erofs.docs.kernel.org/en/latest/) filesystem implementation details
56+
- dm-verity kernel subsystem internals
57+
- IPE policy configuration
58+
59+
**Requirements:**
60+
1. Support for PKCS#7 signature format (in addition to existing JWS/COSE)
61+
2. Per-layer signing capability
62+
3. Deterministic EROFS image and Merkle tree generation
63+
4. OCI registry distribution for a new artifact containing the signed layer root hashes via ORAS Attached Artifacts
64+
65+
### Extended Notation CLI
66+
67+
Extend `notation sign` with a new `--dm-verity` flag to enable automated per-layer signing. While the command below assumes the container image exists in a remote registry, this argument should also work when signing [OCI image layouts](https://github.com/notaryproject/notation/blob/main/specs/cmd/sign.md#experimental-sign-container-images-stored-in-oci-layout-directory) with argument `--oci-layout` for local signing.
68+
69+
The manifest from the default sign behavior is signed with the expected JWS/COSE formats that can be verified in userspace while the layer hashes are signed with the PKCS#7 format by default until other formats are supported.
70+
71+
This command will not recursively sign multi-arch container images. In this case, the command should be run for each individual image for the requested architecture.
72+
73+
**Sample command:**
74+
```bash
75+
notation sign --dm-verity \
76+
--id myKeyId \
77+
myregistry.azurecr.io/myapp@sha256:def456...
78+
```
79+
**Sample output:**
80+
```
81+
Successfully signed myregistry.azurecr.io/myapp@sha256:def456...
82+
Pushed the dm-verity signatures to myregistry.azurecr.io/myapp@sha256:439dd2...
83+
```
84+
85+
**Steps:**
86+
87+
1. **Pull the image manifest** from the registry.
88+
2. **Iterate through all layers** in the manifest. For each layer:
89+
- Pull the layer blob
90+
- Generate an EROFS image by decompressing the tar layer and using `mkfs.erofs` (deterministic, read-only filesystem image)
91+
- Compute the dm-verity Merkle tree root hash from the EROFS image
92+
- Sign the root hash using PKCS#7
93+
3. **Create a signature envelope** for each layer containing:
94+
- The signed root hash of the EROFS image.
95+
- Signer cert embedded inside the PKCS#7 signature blob
96+
- Digest info of the original layer digest blob
97+
- Digest and size of the PKCS#7 signature file
98+
4. **Create a signature manifest** containing:
99+
- All per-layer signature envelopes
100+
5. **Attach the signature manifest** to the image manifest as an attached artifact (referrer) in the registry.
101+
102+
**Example signature manifest structure:**
103+
104+
```json
105+
{
106+
"schemaVersion": 2,
107+
"mediaType": "application/vnd.oci.image.manifest.v1+json",
108+
"artifactType": "application/vnd.cncf.notary.signature.dm-verity",
109+
"config": {
110+
"mediaType": "application/vnd.oci.empty.v1+json",
111+
"digest": "sha256:44136fa355b3678a1146ad16f7e8649e94fb4fc21fe77e8310c060f61caaff8a",
112+
"size": 2
113+
},
114+
"layers": [
115+
{
116+
"mediaType": "application/pkcs7-signature",
117+
"digest": "sha256:abc123...",
118+
"size": 1479,
119+
"annotations": {
120+
"io.cncf.notary.layer.digest": "sha256:layer0digest...",
121+
"io.cncf.notary.dm-verity.root-hash": "0dcd29977f675344645e8c907b5a86b490335e7a2657a2ba45d00e7944701eed",
122+
"org.opencontainers.image.title": "layer-0.pkcs7.sig"
123+
}
124+
}
125+
],
126+
"subject": {
127+
"mediaType": "application/vnd.oci.image.manifest.v1+json",
128+
"digest": "sha256:imagemanifestdigest...",
129+
"size": 1234
130+
},
131+
"annotations": {
132+
"io.cncf.notary.dm-verity.signature": "true"
133+
}
134+
}
135+
```
136+
137+
The layers section now contains metadata for each image layer signature. Each entry describes:
138+
- mediaType: The format of the signature (PKCS#7).
139+
- digest: The SHA-256 hash of the signature file.
140+
- size: The signature file size in bytes.
141+
- annotations: Extra metadata, including:
142+
- The digest of the signed layer.
143+
- The dm-verity root hash for integrity verification.
144+
- A human-readable title for the signature file.
145+
146+
The new entries are described below:
147+
- io.cncf.notary.layer.digest: The digest of the original image layer
148+
- io.cncf.notary.dm-verity.root-hash: The root hash value of the dm-verity block device
149+
- io.cncf.notary.dm-verity.signature=true: This is a flag that notifies Notation that dm-verity signatures and root hashes exist in the artifact
150+
151+
**Performance Metrics:**
152+
- Registry overhead: ~4 KB per layer
153+
- PKCS#7 signature blob: ~2 KB
154+
- Manifest entry: ~2 KB
155+
- Signing time: ~4-5 seconds per layer
156+
- Timeout: 5 minutes for EROFS conversion with no hardcoded maximum layer size
157+
158+
159+
**Verification command:**
160+
161+
```bash
162+
notation verify myregistry.azurecr.io/myapp@sha256:def456...
163+
```
164+
165+
The command interface will not change. This command will not check layer signature compatibility with any keys. It will only output additional information for dm-verity. If the image has dm-verity signatures attached, Notation should:
166+
1. Detect the `io.cncf.notary.dm-verity.signature=true` annotation
167+
2. Inform the user that kernel-level verification is required
168+
169+
**Sample output:**
170+
171+
```
172+
Successfully verified signature for myregistry.azurecr.io/myapp@sha256:def456...
173+
Note: This image includes dm-verity layer signatures for kernel-enforced integrity.
174+
```
175+
176+
### PKCS#7 Signature Format Support
177+
178+
To enable kernel verification, Notation must support PKCS#7 signature envelopes:
179+
180+
**Requirements:**
181+
- PKCS#7 envelope generation for signing
182+
- X.509 certificate chain embedding
183+
- Compatibility with Linux kernel key rings
184+
- User-mode verification support (for build-time validation)
185+
186+
187+
### Runtime Verification Workflow (implemented in containerd)
188+
189+
This work is ongoing in the containerd project and is described in milestone 1 of the [RFC for Code Integrity](https://github.com/containerd/containerd/issues/12081). The EROFS containerd snapshotter implements the following:
190+
191+
1. **Container Start**: When a container is scheduled, fetch the image manifest
192+
2. **Signature Discovery**: Fetch the signature referrer artifact if one is attached to the image manifest
193+
3. **Layer Processing**: For each layer:
194+
- Fetch the PKCS#7 signature and metadata from the signature manifest
195+
- Generate the EROFS image from the decompressed tar file
196+
- Compute the dm-verity Merkle tree root hash from the layer content
197+
- Create a dm-verity block device, passing the root hash and PKCS#7 signature to the kernel
198+
4. **Kernel Verification**: The kernel verifies the PKCS#7 signature against the trusted keyring
199+
5. **Mount**: If verification succeeds, the kernel mounts the dm-verity protected layer
200+
6. **IPE Enforcement**: IPE policies allow code execution only from verified dm-verity volumes
201+
202+
This provides continuous integrity protection. Any tampering with layer content causes dm-verity verification to fail, preventing mount and execution.

0 commit comments

Comments
 (0)